workflow: require acceptance tests in auto-fix PRs#16039
Merged
Conversation
The triage workflow was producing PRs with only unit tests (mocks), missing end-to-end coverage that would catch integration failures. Add explicit testing requirements: every fix must include an acceptance test unless the scenario is already covered by an existing one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the issue repro triage auto-fix workflow guidance to require end-to-end acceptance coverage for automated fixes, addressing gaps where unit tests alone missed real pipeline behavior.
Changes:
- Replaces the generic test-writing step with explicit testing requirements.
- Adds acceptance-test criteria, checklist, and guidance for when existing coverage is sufficient.
| **How to decide:** | ||
| 1. Search `test/Microsoft.TestPlatform.Acceptance.IntegrationTests/` for existing tests covering the same scenario (same framework, same feature, same failure mode). | ||
| 2. If an existing test already exercises the exact code path that was broken → add a comment in the PR noting which test covers it. No new acceptance test needed. | ||
| 3. If no existing test covers it → write one. Follow the patterns in the acceptance test project (test assets under `test/TestAssets/`, `AcceptanceTestBase` helpers, `InvokeVsTestConsole`/`InvokeDotnetTest` for execution). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The triage workflow was producing PRs with only unit tests (mocks), missing end-to-end coverage. For example, #16033 has good unit tests but no acceptance test proving the data-driven scenario works through the real pipeline.
Adds explicit testing requirements to Step 4b: every fix must include an acceptance test unless the scenario is already covered by an existing one. Includes guidance on where acceptance tests live, how to decide if one is needed, and a checklist.