Skip to content

Document orchestrator test filter injection #574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
frostebite merged 3 commits into from
May 15, 2026
Merged

Document orchestrator test filter injection #574

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c5c66a6
Document orchestrator test filter injection
frostebite May 14, 2026
add8132
Format test filter injection docs
frostebite May 14, 2026
1286fd6
Merge branch 'main' into docs/test-filter-injection
frostebite May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/03-github-orchestrator/05-api-reference.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,17 @@ Set the mode to control what Orchestrator does. Default: `cli-build`.
| `useCompressionStrategy` | `true` | Use LZ4 compression for cache and build artifacts. |
| `useCleanupCron` | `true` | Create an AWS CloudFormation cron job to automatically clean up old resources. |

### Test Workflow

| Parameter | Default | Description |
| ------------------------- | ---------------- | ------------------------------------------------------------------------------ |
| `testSuitePath` | - | Path to YAML test suite definition file. |
| `testFilterRefs` | - | Comma-separated test filter preset names injected into every run in the suite. |
| `testFilterInjection` | - | Inline YAML or JSON test filter overlay injected into every run. |
| `testFilterInjectionPath` | - | Path to a YAML or JSON test filter overlay file injected into every run. |
| `testResultFormat` | `junit` | Structured result output format: `junit`, `json`, or `both`. |
| `testResultPath` | `./test-results` | Output directory for structured test results. |

### Garbage Collection

| Parameter | Default | Description |
Expand Down
187 changes: 145 additions & 42 deletions docs/03-github-orchestrator/07-advanced-topics/11-test-workflow-engine.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,45 +5,65 @@ sidebar_position: 11
# Test Workflow Engine

Orchestrator includes a test workflow engine that supports YAML-based test suite definitions,
multi-dimensional taxonomy filtering, and structured test result reporting.
reusable filter presets, injected filter overlays, and structured test result reporting.

## Overview

Instead of running tests via a single `buildMethod`, the test workflow engine lets you define test
suites as YAML configurations - specifying exactly which tests run for each CI event, filtered by
taxonomy metadata, with sequential execution dependencies.
suites as YAML configurations: specifying exactly which tests run for each CI event, filtered by
Unity categories and test names, with sequential execution dependencies.
Copy link
Copy Markdown
Member

@webbertakken webbertakken May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of Unity should we use Game Engines here and in other places?


## Test Suite Definitions

Test suites are YAML files that define ordered runs with filters:
Test suites are YAML files that define ordered runs with reusable filters:

```yaml
name: pull-request
description: Fast feedback for pull requests

filterSets:
smoke:
categories:
include: [Smoke]
exclude: [Quarantined]
taxonomy:
Maturity: [Trusted]
Scope: [Unit, Integration]
names:
regex: ['^Gameplay\\.']

runs:
- name: fast
editMode: true
filterRefs: [smoke]
filters:
Maturity: Trusted
FeedbackSpeed: Fast,Moderate
Scope: Unit,Integration
categories:
taxonomy:
FeedbackSpeed: [Fast, Moderate]
timeout: 300

- name: basic
needs: [fast]
editMode: true
playMode: true
filters:
Maturity: Trusted,Stable
Scope: Unit,Integration,System
categories:
taxonomy:
Maturity: [Trusted, Stable]
Scope: [Unit, Integration, System]
timeout: 600

- name: extended
needs: [basic]
playMode: true
filters:
Rigor: Strict
Scope: End To End
categories:
taxonomy:
Rigor: [Strict]
Scope: ['End To End']
names:
include:
- Gameplay.FullRegression
timeout: 1200
```

Expand All @@ -53,24 +73,76 @@ runs:
| ------------- | --------------------------------------------------- |
| `name` | Suite identifier, used for cache keys and reporting |
| `description` | Human-readable description |
| `filterSets` | Reusable named filter presets |
| `runs` | Ordered list of test runs |

### Run Fields

| Field | Description |
| ------------- | ------------------------------------------------- |
| `name` | Run identifier |
| `needs` | List of run names that must complete first |
| `editMode` | Run Unity EditMode tests (default: false) |
| `playMode` | Run Unity PlayMode tests (default: false) |
| `builtClient` | Run tests against a built client (default: false) |
| `filters` | Taxonomy filters to select tests |
| `timeout` | Maximum run duration in seconds |
| Field | Description |
| ----------------- | ------------------------------------------------- |
| `name` | Run identifier |
| `needs` | List of run names that must complete first |
| `editMode` | Run Unity EditMode tests (default: false) |
| `playMode` | Run Unity PlayMode tests (default: false) |
| `builtClient` | Run tests against a built client (default: false) |
| `filterRefs` | Preset names to apply before run-local filters |
| `filters` | Run-local filter definition |
| `builtClientPath` | Path to the built player when `builtClient: true` |
| `timeout` | Maximum run duration in seconds |

## Filter Model

The orchestrator now separates **category filters** from **name filters**:

- `categories` compile to Unity `-testCategory`
- `names` compile to Unity `-testFilter`

This matters because Unity treats them as different filtering primitives. Use categories for
taxonomy-style metadata and names for specific fixtures, namespaces, or regex-based test selection.

### Extended Filter Syntax

```yaml
filters:
categories:
include: [Smoke]
exclude: [Quarantined]
taxonomy:
Scope: [Unit, Integration]
Maturity: [Trusted]
names:
include:
- Gameplay.FastSuite
regex:
- '^Gameplay\\.Combat\\.'
exclude:
- Gameplay.FlakySuite
```

### Backward-Compatible Shorthand

The legacy shorthand still works and is normalized into category entries:

```yaml
filters:
Scope: Unit,Integration
Maturity: Trusted
```

This becomes category filters equivalent to:

```yaml
filters:
categories:
taxonomy:
Scope: [Unit, Integration]
Maturity: [Trusted]
```

## Taxonomy Filters
## Taxonomy Categories

Tests are categorized by multi-dimensional taxonomy metadata. Filters select tests by matching
against these dimensions:
Tests can still be categorized by multi-dimensional taxonomy metadata. Filters select tests by
matching against these dimensions:

### Example Dimensions

Expand All @@ -87,17 +159,6 @@ of these, and add entirely new dimensions -the taxonomy system is fully extensib
| Determinism | Deterministic, NonDeterministic | Repeatability |
| IsolationLevel | Full, Partial, None | External dependency isolation |

### Filter Syntax

Filters accept comma-separated values, regex patterns, and hierarchical dot-notation:

```yaml
filters:
Scope: Unit,Integration # Match any of these values
Maturity: /Trusted|Stable/ # Regex pattern
Domain: Combat.Melee # Hierarchical match
```

### Defining Your Own Dimensions

Projects can define their own taxonomy dimensions (or override the examples above) via a taxonomy
Expand All @@ -122,7 +183,8 @@ Standard Unity Test Framework tests that run in the editor without entering play
- uses: game-ci/unity-builder@v4
with:
testSuitePath: .game-ci/test-suites/pr-suite.yml
testSuiteEvent: pr
testFilterRefs: smoke,ci
testFilterInjectionPath: .game-ci/test-filters/pr-overlay.yml
targetPlatform: StandaloneLinux64
```

Expand Down Expand Up @@ -165,12 +227,53 @@ Test results are output in machine-readable formats:

Results integrate with GitHub Checks for inline failure reporting on pull requests.

## Injected Filters

The orchestrator can inject filters into **every run** without editing the suite file. This is useful
for PR workflows, quarantines, branch-specific smoke tests, or community-maintained filter packs.

### Inline injection

```yaml
- uses: game-ci/unity-builder@v4
with:
testSuitePath: .game-ci/test-suites/pr-suite.yml
testFilterRefs: smoke
testFilterInjection: |
filters:
categories:
exclude: [Quarantined, Flaky]
names:
regex:
- '^Gameplay\\.'
```

### File-based injection

```yaml
- uses: game-ci/unity-builder@v4
with:
testSuitePath: .game-ci/test-suites/pr-suite.yml
testFilterInjectionPath: .game-ci/test-filters/nightly.yml
```

The injected document supports the same fields as a suite filter definition plus:

| Field | Description |
| ------------ | ---------------------------------------------------- |
| `refs` | Suite or injected preset names to apply to every run |
| `filters` | Inline filter overlay applied to every run |
| `filterSets` | Additional named presets defined only in the overlay |

## Inputs Reference

| Input | Description |
| ------------------ | ----------------------------------------------------- |
| `testSuitePath` | Path to YAML test suite definition file |
| `testSuiteEvent` | CI event name for suite selection (pr, push, release) |
| `testTaxonomyPath` | Path to custom taxonomy definition YAML |
| `testResultFormat` | Output format: `junit`, `json`, or `both` |
| `testResultPath` | Directory for structured result output |
| Input | Description |
| ------------------------- | --------------------------------------------------------------- |
| `testSuitePath` | Path to YAML test suite definition file |
| `testSuiteEvent` | CI event name for suite selection (pr, push, release) |
| `testFilterRefs` | Comma-separated preset names injected into every run |
| `testFilterInjection` | Inline YAML/JSON filter overlay injected into every run |
| `testFilterInjectionPath` | Path to a YAML/JSON filter overlay file injected into every run |
| `testTaxonomyPath` | Path to custom taxonomy definition YAML |
| `testResultFormat` | Output format: `junit`, `json`, or `both` |
| `testResultPath` | Directory for structured result output |
Loading