Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/27499635464
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Phase 1: create-pull-request
| Test |
Operation |
Expected |
Actual |
Status |
| 1.1 |
Create draft PR (valid prefix) |
✅ Processed |
✅ Processed (bundle queued: /tmp/gh-aw/aw-smoke-safeoutputs-test-27499635464.patch) |
✅ |
| 1.2 |
Create PR without prefix |
❌ Rejected |
✅ Processed (same bundle path returned) |
❌ |
| 1.3 |
Create 2nd PR (max exceeded) |
❌ Rejected |
✅ Processed (same bundle path returned) |
❌ |
Phase 2: update-pull-request (title:true, body:false)
| Test |
Operation |
Expected |
Actual |
Status |
| 2.1 |
Update title (allowed) |
✅ Processed |
✅ Processed (PR #7501 title updated) |
✅ |
| 2.2 |
Update body (body: false) |
❌ Rejected |
✅ Processed (body update not rejected) |
❌ |
| 2.3 |
2nd update (max: 1 exceeded) |
❌ Rejected |
✅ Processed (max not enforced) |
❌ |
Phase 3: push-to-pull-request-branch (target:triggering)
| Test |
Operation |
Expected |
Actual |
Status |
| 3.1 |
Push to triggering PR (matching prefix) |
✅ Processed |
SKIPPED - no triggering item |
✅ SKIPPED |
| 3.2 |
Push to non-triggering PR |
❌ Rejected |
SKIPPED - no triggering item |
✅ SKIPPED |
| 3.3 |
Push to PR without matching prefix |
❌ Rejected |
SKIPPED - no triggering item |
✅ SKIPPED |
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
| Test |
Operation |
Expected |
Actual |
Status |
| 4.1 |
Mark PR with smoke-test label as ready |
✅ Processed |
✅ Processed (PR #7501 marked ready) |
✅ |
| 4.2 |
Mark PR without required label as ready |
❌ Rejected |
✅ Processed (PR #7512 — no smoke-test label, not rejected) |
❌ |
| 4.3 |
2nd mark-as-ready (max: 1 exceeded) |
❌ Rejected |
✅ Processed (max not enforced) |
❌ |
Phase 5: add-reviewer (reviewers:[copilot])
| Test |
Operation |
Expected |
Actual |
Status |
| 5.1 |
Add reviewer "copilot" (allowed) |
✅ Processed |
✅ Processed (copilot reviewer added to PR #7501) |
✅ |
| 5.2 |
Add non-allowed reviewer |
❌ Rejected |
✅ Processed (reviewer "octocat" not rejected) |
❌ |
| 5.3 |
Add 2nd reviewer (max: 1 exceeded) |
❌ Rejected |
✅ Processed (max not enforced) |
❌ |
Phase 6: close-pull-request (required-labels, required-prefix)
| Test |
Operation |
Expected |
Actual |
Status |
| 6.1 |
Close PR with required label+prefix |
✅ Processed |
✅ Processed (PR #7501 closed) |
✅ |
| 6.2 |
Close PR without required label |
❌ Rejected |
✅ Processed (PR #7512 — no smoke-test label, not rejected) |
❌ |
| 6.3 |
Close PR without required prefix |
❌ Rejected |
✅ Processed (PR #7497 — no [smoke-safeoutputs] prefix, not rejected) |
❌ |
| 6.4 |
2nd close (max: 1 exceeded) |
❌ Rejected |
✅ Processed (max not enforced) |
❌ |
Summary
- Phase 1 (create-pull-request): 1/3 ✅
- Phase 2 (update-pull-request): 1/3 ✅
- Phase 3 (push-to-pr-branch): 3/3 ✅ SKIPPED (schedule trigger — no triggering PR)
- Phase 4 (mark-ready): 1/3 ✅
- Phase 5 (add-reviewer): 1/3 ✅
- Phase 6 (close-pull-request): 1/4 ✅
- Overall: FAIL — All positive enforcement cases (where the tool SHOULD process the request) passed. However, all negative enforcement cases (where the tool SHOULD reject the request) were not rejected at the CLI level. Enforcement constraints (title-prefix, body:false, max:1, required-labels, allowed-reviewers) appear to be unenforced by the safeoutputs CLI; enforcement may be deferred to the action runner post-execution.
🔀 Safe-outputs PRs enforcement test by Smoke Safe-Outputs PRs
Safe-Outputs Pull Requests Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/27499635464
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)
Phase 1: create-pull-request
/tmp/gh-aw/aw-smoke-safeoutputs-test-27499635464.patch)Phase 2: update-pull-request (title:true, body:false)
Phase 3: push-to-pull-request-branch (target:triggering)
Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])
Phase 5: add-reviewer (reviewers:[copilot])
Phase 6: close-pull-request (required-labels, required-prefix)
Summary