[RFC] Microsoft.Testing.Extensions.GitHubReport: publish test results to GitHub from MTP

[Evangelink]

Summary

Propose a new MTP extension Microsoft.Testing.Extensions.GitHubReport that publishes test results back to GitHub when the test session runs in GitHub Actions. Mirrors the design and CLI shape of the existing Microsoft.Testing.Extensions.AzureDevOpsReport extension.

Motivation

Today, users running MTP-powered test projects on GitHub Actions get test results in two places:

1. The raw stdout of the workflow step.
2. A separate, post-process Action (dorny/test-reporter, EnricoMi/publish-unit-test-result-action, ctrf-io/github-test-reporter, …) that parses TRX or JUnit after the test step finishes.

The post-process path works but:

• It runs after the fact (no live progress, no real-time annotations).
• File/line annotations are best-effort regex over TRX/JUnit, not authoritative test-node metadata.
• It requires every consumer to wire up a separate job step.

We already have Microsoft.Testing.Extensions.AzureDevOpsReport, which integrates natively with Azure DevOps (live publishing, build summary, annotations, progress, artifact upload). The GitHub equivalent is a natural symmetry to add.

Proposed design

Detection

Mirror the AzDO extension''s TF_BUILD=true gating:

• GITHUB_ACTIONS=true enables the extension.
• GITHUB_TOKEN, GITHUB_REPOSITORY, GITHUB_REF, GITHUB_SHA, GITHUB_RUN_ID, GITHUB_STEP_SUMMARY consumed as needed (each tier degrades gracefully if its required vars are missing, with a trace log).
• All CLI options no-op outside GitHub Actions and emit a warning if explicitly set (same pattern as --report-azdo*).

Three tiers (adoptable incrementally)

Tier	Feature	Auth	GitHub API
1	Job summary + workflow logging commands — write a Markdown summary to $GITHUB_STEP_SUMMARY; emit ::error file=…,line=…,title=…::message for failures, ::warning file=… for skipped, ::notice for retries.	None	Built-in (file + stdout)
2	Check Run with annotations — create a Check Run, attach per-failure annotations pinned to file:line using TestFileLocationProperty from the TestNode.	GITHUB_TOKEN	POST/PATCH /repos/{owner}/{repo}/check-runs
3	Sticky PR comment — post/update a single comment on the PR with the summary table (find-or-create by marker).	GITHUB_TOKEN + pull_request event	POST/PATCH /repos/.../issues/{n}/comments

CLI options (mirroring --report-azdo*)

• --report-github — master switch (default on when GITHUB_ACTIONS=true).
• --report-github-summary — Tier 1 Markdown summary.
• --report-github-progress — Tier 1 progress via step summary updates.
• --report-github-annotations — Tier 1 ::error workflow commands (and Tier 2 Check Run annotations if a token is available).
• --report-github-comment — Tier 3 sticky PR comment.

Implementation outline

File layout mirrors src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/:

• GitHubExtensions.cs — public AddGitHubProvider(this ITestApplicationBuilder) entry point.
• GitHubCommandLineProvider.cs / GitHubCommandLineOptions.cs — CLI surface.
• GitHubReporter.cs — env detection, master switch.
• GitHubSummaryReporter.cs — Tier 1 summary writer.
• GitHubAnnotationEmitter.cs — Tier 1 workflow command emitter.
• GitHubCheckRunClient.cs / GitHubCheckRunPublisher.cs — Tier 2 Check Runs API.
• GitHubPullRequestCommentClient.cs / GitHubPullRequestCommentPublisher.cs — Tier 3 sticky comment.
• TestingPlatformBuilderHook.cs — auto-registration hook.
• PACKAGE.md, BannedSymbols.txt, csproj.

Why an MTP extension instead of a separate Action?

Aspect	MTP extension	Post-process Action
Setup	Reference one NuGet + flag	Add a separate workflow step
Live progress	✅ Check Run updates	❌ After test step exits
File/line annotation fidelity	✅ Direct from TestNode metadata	Best-effort regex over TRX
Works for fork PRs	✅ GITHUB_TOKEN	✅
Maintenance	We own it	Third-party

Trade-offs / open questions

1. Doesn''t help the microsoft/testfx repo''s own CI — testfx runs on AzDO, not GitHub Actions, so this extension wouldn''t have GITHUB_TOKEN available in our own pipelines. This is primarily useful for users of MTP running on GitHub Actions.
2. HTTP from the test host — Tier 2/3 introduce REST calls during/after the test session. AzDO extension already does the same; design GitHubCheckRunClient after AzureDevOpsTestResultsClient.
3. Permission scoping — Tier 2 needs checks: write; Tier 3 needs pull-requests: write. Document required permissions: block in PACKAGE.md.
4. Authentication beyond GITHUB_TOKEN — should we also accept a PAT or GitHub App token via env var (e.g. for self-hosted/Jenkins-running-against-GitHub-PR scenarios)? Recommend yes, via GITHUBREPORTER_TOKEN fallback.
5. Annotation aggregation limits — GitHub caps Check Run annotations at 50 per request; need batching for large failure counts (mirroring how AzureDevOpsArtifactUploader batches).
6. xUnit/NUnit/TUnit — should work via MTP for any test framework that emits TestFileLocationProperty; verify and document gaps.

Out of scope

• Auto-creating a PR with fix suggestions.
• Coverage publishing (separate concern; can be a sibling extension or layered on top once the GitHub plumbing exists).
• Trend/history (the AzDO extension''s AzureDevOpsHistoryService is AzDO-API-specific; GitHub equivalent would need workflow-run-history APIs and is a follow-up).

Acceptance criteria

• New extension project under src/Platform/Microsoft.Testing.Extensions.GitHubReport/ builds and packs as Microsoft.Testing.Extensions.GitHubReport.
• Tier 1 works in any GitHub Actions workflow with zero additional configuration.
• Tier 2/3 work when permissions: grants checks: write / pull-requests: write.
• Acceptance tests under test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/ exercise each tier with a fake/stub GitHub server.
• PACKAGE.md documents required workflow permissions: blocks and the CLI surface.

---

🤖 RFC drafted with the assistance of GitHub Copilot CLI.