From cce16317330c3c7c4732e0a350fa64da12322b04 Mon Sep 17 00:00:00 2001 From: Jakub Jares Date: Mon, 18 May 2026 11:17:13 +0200 Subject: [PATCH] workflow: require acceptance tests in auto-fix PRs The triage workflow was producing PRs with only unit tests (mocks), missing end-to-end coverage that would catch integration failures. Add explicit testing requirements: every fix must include an acceptance test unless the scenario is already covered by an existing one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/issue-repro-triage.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/.github/workflows/issue-repro-triage.md b/.github/workflows/issue-repro-triage.md index 3828148cb7..016a3e5fd1 100644 --- a/.github/workflows/issue-repro-triage.md +++ b/.github/workflows/issue-repro-triage.md @@ -169,10 +169,31 @@ Before writing any code, search for existing open PRs that already address this 1. **Read AGENTS.md** for repo conventions 2. **Understand the root cause** by reading the relevant source code 3. **Implement a fix** on a new branch `fix/issue-` -4. **Write a test** that fails without the fix and passes with it +4. **Write tests** — see testing requirements below 5. **Build and run tests** to verify nothing is broken 6. **Create a draft PR** referencing the issue +##### Testing Requirements + +Every fix **must** include an acceptance test (end-to-end) unless the exact scenario is already covered by an existing acceptance test. Unit tests alone are not sufficient. + +**Acceptance tests** live in `test/Microsoft.TestPlatform.Acceptance.IntegrationTests/`. They exercise the real test platform pipeline — building and running actual test projects through `dotnet test` or the translation layer — and verify externally observable behavior (console output, TRX content, exit codes, data collector artifacts, etc.). + +**Why this matters:** Unit tests with mocks verify internal wiring but miss integration failures — wrong event ordering, serialization issues, host process boundaries, adapter interactions. The bug being fixed was already "tested" by internal logic; what was missing is proof that the scenario works end-to-end. + +**How to decide:** +1. Search `test/Microsoft.TestPlatform.Acceptance.IntegrationTests/` for existing tests covering the same scenario (same framework, same feature, same failure mode). +2. If an existing test already exercises the exact code path that was broken → add a comment in the PR noting which test covers it. No new acceptance test needed. +3. If no existing test covers it → write one. Follow the patterns in the acceptance test project (test assets under `test/TestAssets/`, `AcceptanceTestBase` helpers, `InvokeVsTestConsole`/`InvokeDotnetTest` for execution). + +**Acceptance test checklist:** +- [ ] Uses a real test project (existing test asset or new minimal one under `test/TestAssets/`) +- [ ] Runs through the actual test host (not mocked) +- [ ] Asserts on externally visible output (TRX results, stdout, exit code, artifacts) +- [ ] Fails without the fix applied (verify by temporarily reverting the source change) + +Unit tests are still welcome as supplementary coverage for edge cases and internal invariants, but they don't replace the acceptance test requirement. + The PR description should include: - 🤖 disclosure that this is an automated fix - Link to the issue