From c5c66a6a573a93d75b15056b478c56e638688020 Mon Sep 17 00:00:00 2001 From: frostebite Date: Thu, 14 May 2026 23:56:37 +0100 Subject: [PATCH 1/2] Document orchestrator test filter injection --- .../05-api-reference.mdx | 11 ++ .../11-test-workflow-engine.mdx | 187 ++++++++++++++---- 2 files changed, 156 insertions(+), 42 deletions(-) diff --git a/docs/03-github-orchestrator/05-api-reference.mdx b/docs/03-github-orchestrator/05-api-reference.mdx index 2b56d455..271cfa7d 100644 --- a/docs/03-github-orchestrator/05-api-reference.mdx +++ b/docs/03-github-orchestrator/05-api-reference.mdx @@ -135,6 +135,17 @@ Set the mode to control what Orchestrator does. Default: `cli-build`. | `useCompressionStrategy` | `true` | Use LZ4 compression for cache and build artifacts. | | `useCleanupCron` | `true` | Create an AWS CloudFormation cron job to automatically clean up old resources. | +### Test Workflow + +| Parameter | Default | Description | +| ------------------------- | ------------------ | -------------------------------------------------------------------------------------------- | +| `testSuitePath` | - | Path to YAML test suite definition file. | +| `testFilterRefs` | - | Comma-separated test filter preset names injected into every run in the suite. | +| `testFilterInjection` | - | Inline YAML or JSON test filter overlay injected into every run. | +| `testFilterInjectionPath` | - | Path to a YAML or JSON test filter overlay file injected into every run. | +| `testResultFormat` | `junit` | Structured result output format: `junit`, `json`, or `both`. | +| `testResultPath` | `./test-results` | Output directory for structured test results. | + ### Garbage Collection | Parameter | Default | Description | diff --git a/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx b/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx index c6790e26..b4deccfe 100644 --- a/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx +++ b/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx @@ -5,28 +5,41 @@ sidebar_position: 11 # Test Workflow Engine Orchestrator includes a test workflow engine that supports YAML-based test suite definitions, -multi-dimensional taxonomy filtering, and structured test result reporting. +reusable filter presets, injected filter overlays, and structured test result reporting. ## Overview Instead of running tests via a single `buildMethod`, the test workflow engine lets you define test -suites as YAML configurations - specifying exactly which tests run for each CI event, filtered by -taxonomy metadata, with sequential execution dependencies. +suites as YAML configurations: specifying exactly which tests run for each CI event, filtered by +Unity categories and test names, with sequential execution dependencies. ## Test Suite Definitions -Test suites are YAML files that define ordered runs with filters: +Test suites are YAML files that define ordered runs with reusable filters: ```yaml name: pull-request description: Fast feedback for pull requests + +filterSets: + smoke: + categories: + include: [Smoke] + exclude: [Quarantined] + taxonomy: + Maturity: [Trusted] + Scope: [Unit, Integration] + names: + regex: ['^Gameplay\\.'] + runs: - name: fast editMode: true + filterRefs: [smoke] filters: - Maturity: Trusted - FeedbackSpeed: Fast,Moderate - Scope: Unit,Integration + categories: + taxonomy: + FeedbackSpeed: [Fast, Moderate] timeout: 300 - name: basic @@ -34,16 +47,23 @@ runs: editMode: true playMode: true filters: - Maturity: Trusted,Stable - Scope: Unit,Integration,System + categories: + taxonomy: + Maturity: [Trusted, Stable] + Scope: [Unit, Integration, System] timeout: 600 - name: extended needs: [basic] playMode: true filters: - Rigor: Strict - Scope: End To End + categories: + taxonomy: + Rigor: [Strict] + Scope: ['End To End'] + names: + include: + - Gameplay.FullRegression timeout: 1200 ``` @@ -53,24 +73,76 @@ runs: | ------------- | --------------------------------------------------- | | `name` | Suite identifier, used for cache keys and reporting | | `description` | Human-readable description | +| `filterSets` | Reusable named filter presets | | `runs` | Ordered list of test runs | ### Run Fields -| Field | Description | -| ------------- | ------------------------------------------------- | -| `name` | Run identifier | -| `needs` | List of run names that must complete first | -| `editMode` | Run Unity EditMode tests (default: false) | -| `playMode` | Run Unity PlayMode tests (default: false) | -| `builtClient` | Run tests against a built client (default: false) | -| `filters` | Taxonomy filters to select tests | -| `timeout` | Maximum run duration in seconds | +| Field | Description | +| --------------- | ------------------------------------------------- | +| `name` | Run identifier | +| `needs` | List of run names that must complete first | +| `editMode` | Run Unity EditMode tests (default: false) | +| `playMode` | Run Unity PlayMode tests (default: false) | +| `builtClient` | Run tests against a built client (default: false) | +| `filterRefs` | Preset names to apply before run-local filters | +| `filters` | Run-local filter definition | +| `builtClientPath` | Path to the built player when `builtClient: true` | +| `timeout` | Maximum run duration in seconds | + +## Filter Model + +The orchestrator now separates **category filters** from **name filters**: + +- `categories` compile to Unity `-testCategory` +- `names` compile to Unity `-testFilter` + +This matters because Unity treats them as different filtering primitives. Use categories for +taxonomy-style metadata and names for specific fixtures, namespaces, or regex-based test selection. + +### Extended Filter Syntax + +```yaml +filters: + categories: + include: [Smoke] + exclude: [Quarantined] + taxonomy: + Scope: [Unit, Integration] + Maturity: [Trusted] + names: + include: + - Gameplay.FastSuite + regex: + - '^Gameplay\\.Combat\\.' + exclude: + - Gameplay.FlakySuite +``` + +### Backward-Compatible Shorthand + +The legacy shorthand still works and is normalized into category entries: + +```yaml +filters: + Scope: Unit,Integration + Maturity: Trusted +``` + +This becomes category filters equivalent to: + +```yaml +filters: + categories: + taxonomy: + Scope: [Unit, Integration] + Maturity: [Trusted] +``` -## Taxonomy Filters +## Taxonomy Categories -Tests are categorized by multi-dimensional taxonomy metadata. Filters select tests by matching -against these dimensions: +Tests can still be categorized by multi-dimensional taxonomy metadata. Filters select tests by +matching against these dimensions: ### Example Dimensions @@ -87,17 +159,6 @@ of these, and add entirely new dimensions -the taxonomy system is fully extensib | Determinism | Deterministic, NonDeterministic | Repeatability | | IsolationLevel | Full, Partial, None | External dependency isolation | -### Filter Syntax - -Filters accept comma-separated values, regex patterns, and hierarchical dot-notation: - -```yaml -filters: - Scope: Unit,Integration # Match any of these values - Maturity: /Trusted|Stable/ # Regex pattern - Domain: Combat.Melee # Hierarchical match -``` - ### Defining Your Own Dimensions Projects can define their own taxonomy dimensions (or override the examples above) via a taxonomy @@ -122,7 +183,8 @@ Standard Unity Test Framework tests that run in the editor without entering play - uses: game-ci/unity-builder@v4 with: testSuitePath: .game-ci/test-suites/pr-suite.yml - testSuiteEvent: pr + testFilterRefs: smoke,ci + testFilterInjectionPath: .game-ci/test-filters/pr-overlay.yml targetPlatform: StandaloneLinux64 ``` @@ -165,12 +227,53 @@ Test results are output in machine-readable formats: Results integrate with GitHub Checks for inline failure reporting on pull requests. +## Injected Filters + +The orchestrator can inject filters into **every run** without editing the suite file. This is useful +for PR workflows, quarantines, branch-specific smoke tests, or community-maintained filter packs. + +### Inline injection + +```yaml +- uses: game-ci/unity-builder@v4 + with: + testSuitePath: .game-ci/test-suites/pr-suite.yml + testFilterRefs: smoke + testFilterInjection: | + filters: + categories: + exclude: [Quarantined, Flaky] + names: + regex: + - '^Gameplay\\.' +``` + +### File-based injection + +```yaml +- uses: game-ci/unity-builder@v4 + with: + testSuitePath: .game-ci/test-suites/pr-suite.yml + testFilterInjectionPath: .game-ci/test-filters/nightly.yml +``` + +The injected document supports the same fields as a suite filter definition plus: + +| Field | Description | +| ------------ | -------------------------------------------------------- | +| `refs` | Suite or injected preset names to apply to every run | +| `filters` | Inline filter overlay applied to every run | +| `filterSets` | Additional named presets defined only in the overlay | + ## Inputs Reference -| Input | Description | -| ------------------ | ----------------------------------------------------- | -| `testSuitePath` | Path to YAML test suite definition file | -| `testSuiteEvent` | CI event name for suite selection (pr, push, release) | -| `testTaxonomyPath` | Path to custom taxonomy definition YAML | -| `testResultFormat` | Output format: `junit`, `json`, or `both` | -| `testResultPath` | Directory for structured result output | +| Input | Description | +| ------------------------- | ----------------------------------------------------------------- | +| `testSuitePath` | Path to YAML test suite definition file | +| `testSuiteEvent` | CI event name for suite selection (pr, push, release) | +| `testFilterRefs` | Comma-separated preset names injected into every run | +| `testFilterInjection` | Inline YAML/JSON filter overlay injected into every run | +| `testFilterInjectionPath` | Path to a YAML/JSON filter overlay file injected into every run | +| `testTaxonomyPath` | Path to custom taxonomy definition YAML | +| `testResultFormat` | Output format: `junit`, `json`, or `both` | +| `testResultPath` | Directory for structured result output | From add81326571bd7b157a3e8ca5f456ad233947474 Mon Sep 17 00:00:00 2001 From: frostebite Date: Fri, 15 May 2026 00:01:07 +0100 Subject: [PATCH 2/2] Format test filter injection docs --- .../05-api-reference.mdx | 16 +++--- .../11-test-workflow-engine.mdx | 50 +++++++++---------- 2 files changed, 33 insertions(+), 33 deletions(-) diff --git a/docs/03-github-orchestrator/05-api-reference.mdx b/docs/03-github-orchestrator/05-api-reference.mdx index 271cfa7d..4b070b37 100644 --- a/docs/03-github-orchestrator/05-api-reference.mdx +++ b/docs/03-github-orchestrator/05-api-reference.mdx @@ -137,14 +137,14 @@ Set the mode to control what Orchestrator does. Default: `cli-build`. ### Test Workflow -| Parameter | Default | Description | -| ------------------------- | ------------------ | -------------------------------------------------------------------------------------------- | -| `testSuitePath` | - | Path to YAML test suite definition file. | -| `testFilterRefs` | - | Comma-separated test filter preset names injected into every run in the suite. | -| `testFilterInjection` | - | Inline YAML or JSON test filter overlay injected into every run. | -| `testFilterInjectionPath` | - | Path to a YAML or JSON test filter overlay file injected into every run. | -| `testResultFormat` | `junit` | Structured result output format: `junit`, `json`, or `both`. | -| `testResultPath` | `./test-results` | Output directory for structured test results. | +| Parameter | Default | Description | +| ------------------------- | ---------------- | ------------------------------------------------------------------------------ | +| `testSuitePath` | - | Path to YAML test suite definition file. | +| `testFilterRefs` | - | Comma-separated test filter preset names injected into every run in the suite. | +| `testFilterInjection` | - | Inline YAML or JSON test filter overlay injected into every run. | +| `testFilterInjectionPath` | - | Path to a YAML or JSON test filter overlay file injected into every run. | +| `testResultFormat` | `junit` | Structured result output format: `junit`, `json`, or `both`. | +| `testResultPath` | `./test-results` | Output directory for structured test results. | ### Garbage Collection diff --git a/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx b/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx index b4deccfe..902f45bb 100644 --- a/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx +++ b/docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx @@ -78,17 +78,17 @@ runs: ### Run Fields -| Field | Description | -| --------------- | ------------------------------------------------- | -| `name` | Run identifier | -| `needs` | List of run names that must complete first | -| `editMode` | Run Unity EditMode tests (default: false) | -| `playMode` | Run Unity PlayMode tests (default: false) | -| `builtClient` | Run tests against a built client (default: false) | -| `filterRefs` | Preset names to apply before run-local filters | -| `filters` | Run-local filter definition | +| Field | Description | +| ----------------- | ------------------------------------------------- | +| `name` | Run identifier | +| `needs` | List of run names that must complete first | +| `editMode` | Run Unity EditMode tests (default: false) | +| `playMode` | Run Unity PlayMode tests (default: false) | +| `builtClient` | Run tests against a built client (default: false) | +| `filterRefs` | Preset names to apply before run-local filters | +| `filters` | Run-local filter definition | | `builtClientPath` | Path to the built player when `builtClient: true` | -| `timeout` | Maximum run duration in seconds | +| `timeout` | Maximum run duration in seconds | ## Filter Model @@ -259,21 +259,21 @@ for PR workflows, quarantines, branch-specific smoke tests, or community-maintai The injected document supports the same fields as a suite filter definition plus: -| Field | Description | -| ------------ | -------------------------------------------------------- | -| `refs` | Suite or injected preset names to apply to every run | -| `filters` | Inline filter overlay applied to every run | -| `filterSets` | Additional named presets defined only in the overlay | +| Field | Description | +| ------------ | ---------------------------------------------------- | +| `refs` | Suite or injected preset names to apply to every run | +| `filters` | Inline filter overlay applied to every run | +| `filterSets` | Additional named presets defined only in the overlay | ## Inputs Reference -| Input | Description | -| ------------------------- | ----------------------------------------------------------------- | -| `testSuitePath` | Path to YAML test suite definition file | -| `testSuiteEvent` | CI event name for suite selection (pr, push, release) | -| `testFilterRefs` | Comma-separated preset names injected into every run | -| `testFilterInjection` | Inline YAML/JSON filter overlay injected into every run | -| `testFilterInjectionPath` | Path to a YAML/JSON filter overlay file injected into every run | -| `testTaxonomyPath` | Path to custom taxonomy definition YAML | -| `testResultFormat` | Output format: `junit`, `json`, or `both` | -| `testResultPath` | Directory for structured result output | +| Input | Description | +| ------------------------- | --------------------------------------------------------------- | +| `testSuitePath` | Path to YAML test suite definition file | +| `testSuiteEvent` | CI event name for suite selection (pr, push, release) | +| `testFilterRefs` | Comma-separated preset names injected into every run | +| `testFilterInjection` | Inline YAML/JSON filter overlay injected into every run | +| `testFilterInjectionPath` | Path to a YAML/JSON filter overlay file injected into every run | +| `testTaxonomyPath` | Path to custom taxonomy definition YAML | +| `testResultFormat` | Output format: `junit`, `json`, or `both` | +| `testResultPath` | Directory for structured result output |