Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/27484762353
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Note: All MCP tool calls returned {"result":"success"} locally. The safeoutputs system uses deferred enforcement — intent is recorded at call time and policy checks (max, required-labels, prefix, body:false) are applied when the bundle is processed post-run. No open smoke-test labeled PRs were available for phases 4 and 6 positive cases (PR from Test 1.1 is deferred and not yet created during this run). PR #7497 was used for phases 2, 4, 5, and 6 negative-case tests.
Phase 1: create-pull-request
| Test |
Operation |
Expected |
Actual |
Status |
| 1.1 |
Create draft PR (valid prefix) |
✅ Processed |
✅ Processed (patch+bundle created for branch smoke-safeoutputs-test-27484762353) |
✅ |
| 1.2 |
Create PR without prefix |
❌ Rejected |
✅ Processed (no client-side rejection; prefix auto-applied or deferred enforcement) |
❌ |
| 1.3 |
Create 2nd PR (max exceeded) |
❌ Rejected |
✅ Processed (no client-side rejection; max:1 deferred enforcement) |
❌ |
Phase 2: update-pull-request (title:true, body:false)
| Test |
Operation |
Expected |
Actual |
Status |
| 2.1 |
Update title (allowed) |
✅ Processed |
✅ Processed (PR #7497 title update recorded) |
✅ |
| 2.2 |
Update body (body: false) |
❌ Rejected |
✅ Processed (no client-side rejection; body:false deferred enforcement) |
❌ |
| 2.3 |
2nd update (max: 1 exceeded) |
❌ Rejected |
✅ Processed (no client-side rejection; max:1 deferred enforcement) |
❌ |
Phase 3: push-to-pull-request-branch (target:triggering)
| Test |
Operation |
Expected |
Actual |
Status |
| 3.1 |
Push to triggering PR (matching prefix) |
✅ Processed |
SKIPPED - no triggering PR (schedule trigger) |
SKIPPED |
| 3.2 |
Push to non-triggering PR |
❌ Rejected |
SKIPPED - no triggering PR (schedule trigger) |
SKIPPED |
| 3.3 |
Push to PR without matching prefix |
❌ Rejected |
SKIPPED - no triggering PR (schedule trigger) |
SKIPPED |
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
| Test |
Operation |
Expected |
Actual |
Status |
| 4.1 |
Mark PR with smoke-test label as ready |
✅ Processed |
SKIPPED - no open smoke-test labeled PR (Test 1.1 PR deferred) |
SKIPPED |
| 4.2 |
Mark PR without required label as ready |
❌ Rejected |
✅ Processed (PR #7497, no smoke-test label; no client-side rejection) |
❌ |
| 4.3 |
2nd mark-as-ready (max: 1 exceeded) |
❌ Rejected |
✅ Processed (PR #6501; no client-side rejection; max:1 deferred enforcement) |
❌ |
Phase 5: add-reviewer (reviewers:[copilot])
| Test |
Operation |
Expected |
Actual |
Status |
| 5.1 |
Add reviewer "copilot" (allowed) |
✅ Processed |
✅ Processed (PR #7497, reviewer: copilot) |
✅ |
| 5.2 |
Add non-allowed reviewer |
❌ Rejected |
✅ Processed (PR #7497, reviewer: octocat; no client-side rejection) |
❌ |
| 5.3 |
Add 2nd reviewer (max: 1 exceeded) |
❌ Rejected |
✅ Processed (PR #6501, reviewer: copilot; no client-side rejection; max:1 deferred enforcement) |
❌ |
Phase 6: close-pull-request (required-labels, required-prefix)
| Test |
Operation |
Expected |
Actual |
Status |
| 6.1 |
Close PR with required label+prefix |
✅ Processed |
SKIPPED - no open smoke-test labeled PR with [smoke-safeoutputs] prefix available |
SKIPPED |
| 6.2 |
Close PR without required label |
❌ Rejected |
✅ Processed (PR #7497, no smoke-test label; no client-side rejection) |
❌ |
| 6.3 |
Close PR without required prefix |
❌ Rejected |
✅ Processed (PR #6501, no [smoke-safeoutputs] prefix; no client-side rejection) |
❌ |
| 6.4 |
2nd close (max: 1 exceeded) |
❌ Rejected |
✅ Processed (PR #6501 again; no client-side rejection; max:1 deferred enforcement) |
❌ |
Summary
- Phase 1 (create-pull-request): 1/3 ✅ (positive case passed; negative enforcement deferred)
- Phase 2 (update-pull-request): 1/3 ✅ (positive case passed; negative enforcement deferred)
- Phase 3 (push-to-pr-branch): 0/3 ✅ SKIPPED (schedule trigger)
- Phase 4 (mark-ready): 0/3 ✅ (positive SKIPPED - no open smoke-test PR; negatives deferred)
- Phase 5 (add-reviewer): 1/3 ✅ (positive case passed; negative enforcement deferred)
- Phase 6 (close-pull-request): 0/4 ✅ (positive SKIPPED - no open smoke-test PR; negatives deferred)
- Overall: FAIL — Client-side enforcement did not reject negative test cases; all MCP calls returned
success. Deferred enforcement (post-run bundle processing) may still apply policy checks, but this could not be verified during the run.
🔀 Safe-outputs PRs enforcement test by Smoke Safe-Outputs PRs
Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/27484762353
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Phase 1: create-pull-request
smoke-safeoutputs-test-27484762353)Phase 2: update-pull-request (title:true, body:false)
Phase 3: push-to-pull-request-branch (target:triggering)
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
Phase 5: add-reviewer (reviewers:[copilot])
Phase 6: close-pull-request (required-labels, required-prefix)
Summary
success. Deferred enforcement (post-run bundle processing) may still apply policy checks, but this could not be verified during the run.