Skip to content

workflow: require acceptance tests in auto-fix PRs #16039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nohwnd merged 1 commit into from
May 18, 2026
Merged

workflow: require acceptance tests in auto-fix PRs #16039

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .github/workflows/issue-repro-triage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,10 +169,31 @@ Before writing any code, search for existing open PRs that already address this
1. **Read AGENTS.md** for repo conventions
2. **Understand the root cause** by reading the relevant source code
3. **Implement a fix** on a new branch `fix/issue-<number>`
4. **Write a test** that fails without the fix and passes with it
4. **Write tests** — see testing requirements below
5. **Build and run tests** to verify nothing is broken
6. **Create a draft PR** referencing the issue

##### Testing Requirements

Every fix **must** include an acceptance test (end-to-end) unless the exact scenario is already covered by an existing acceptance test. Unit tests alone are not sufficient.

**Acceptance tests** live in `test/Microsoft.TestPlatform.Acceptance.IntegrationTests/`. They exercise the real test platform pipeline — building and running actual test projects through `dotnet test` or the translation layer — and verify externally observable behavior (console output, TRX content, exit codes, data collector artifacts, etc.).

**Why this matters:** Unit tests with mocks verify internal wiring but miss integration failures — wrong event ordering, serialization issues, host process boundaries, adapter interactions. The bug being fixed was already "tested" by internal logic; what was missing is proof that the scenario works end-to-end.

**How to decide:**
1. Search `test/Microsoft.TestPlatform.Acceptance.IntegrationTests/` for existing tests covering the same scenario (same framework, same feature, same failure mode).
2. If an existing test already exercises the exact code path that was broken → add a comment in the PR noting which test covers it. No new acceptance test needed.
3. If no existing test covers it → write one. Follow the patterns in the acceptance test project (test assets under `test/TestAssets/`, `AcceptanceTestBase` helpers, `InvokeVsTestConsole`/`InvokeDotnetTest` for execution).

**Acceptance test checklist:**
- [ ] Uses a real test project (existing test asset or new minimal one under `test/TestAssets/`)
- [ ] Runs through the actual test host (not mocked)
- [ ] Asserts on externally visible output (TRX results, stdout, exit code, artifacts)
- [ ] Fails without the fix applied (verify by temporarily reverting the source change)

Unit tests are still welcome as supplementary coverage for edge cases and internal invariants, but they don't replace the acceptance test requirement.

The PR description should include:
- 🤖 disclosure that this is an automated fix
- Link to the issue
Expand Down
Loading