Skip to content

yaml-tests.md

Latest commit

 

History

History
74 lines (53 loc) · 2.29 KB

yaml-tests.md

File metadata and controls

74 lines (53 loc) · 2.29 KB

YAML Tests

FinalRun tests are plain YAML files stored under .finalrun/tests/. Each file defines a single test scenario with natural-language steps that the AI agent executes on a real device or emulator.

Test Fields

Field Required Description
name yes Stable identifier for the scenario
description no Short human-readable summary
setup no Actions to prepare clean state before the test runs
steps yes Ordered natural-language steps executed by the agent
expected_state no Expected UI state after all steps complete

Example

name: login_smoke
description: Verify that a user can log in and reach the home screen.

setup:
  - Clear app data.

steps:
  - Launch the app.
  - Enter ${secrets.email} on the login screen.
  - Enter ${secrets.password} on the password screen.
  - Tap the login button.

expected_state:
  - The home screen is visible.
  - The user's name appears in the header.

Environment Placeholders

Tests can reference dynamic values using placeholder syntax:

  • ${secrets.*} — resolves from OS environment variables and workspace-root .env / .env.<name> files. Use for credentials and sensitive values.
  • ${variables.*} — resolves from non-sensitive values declared in .finalrun/env/*.yaml. Use for locale, feature flags, etc.

Both must be declared in .finalrun/env/<name>.yaml. See environment.md for the full guide on load order and dotenv files.

Suite Manifests

Suite manifests live under .finalrun/suites/ and group test files into logical collections. Each entry resolves under .finalrun/tests/.

name: auth_smoke
description: Covers the authentication smoke scenarios.
tests:
  - auth/login.yaml
  - auth/logout.yaml

Suite Fields

Field Required Description
name yes Suite identifier
description no Short human-readable summary
tests yes List of test file paths (relative to .finalrun/tests/)

Running

# Run a single test
finalrun test auth/login.yaml --platform android --model google/gemini-3-flash-preview

# Run a suite
finalrun suite auth_smoke.yaml --platform android --model google/gemini-3-flash-preview

See cli-reference.md for all available flags.