Skip to content

[smoke-safeoutputs] Smoke Safe-Outputs PRs: 27499635464 #7521

@github-actions

Description

@github-actions

Safe-Outputs Pull Requests Enforcement Test Results

Run: https://github.com/github/gh-aw-mcpg/actions/runs/27499635464
Trigger: schedule
Configuration: create-pull-request (max:1, prefix, draft:true), close-pull-request (required-labels, required-prefix, max:1), update-pull-request (title:true, body:false, max:1), push-to-pr-branch (target:triggering, prefix), mark-ready (required-labels:[smoke-test], max:1), add-reviewer (reviewers:[copilot], max:1)

Phase 1: create-pull-request

Test Operation Expected Actual Status
1.1 Create draft PR (valid prefix) ✅ Processed ✅ Processed (bundle queued: /tmp/gh-aw/aw-smoke-safeoutputs-test-27499635464.patch)
1.2 Create PR without prefix ❌ Rejected ✅ Processed (same bundle path returned)
1.3 Create 2nd PR (max exceeded) ❌ Rejected ✅ Processed (same bundle path returned)

Phase 2: update-pull-request (title:true, body:false)

Test Operation Expected Actual Status
2.1 Update title (allowed) ✅ Processed ✅ Processed (PR #7501 title updated)
2.2 Update body (body: false) ❌ Rejected ✅ Processed (body update not rejected)
2.3 2nd update (max: 1 exceeded) ❌ Rejected ✅ Processed (max not enforced)

Phase 3: push-to-pull-request-branch (target:triggering)

Test Operation Expected Actual Status
3.1 Push to triggering PR (matching prefix) ✅ Processed SKIPPED - no triggering item ✅ SKIPPED
3.2 Push to non-triggering PR ❌ Rejected SKIPPED - no triggering item ✅ SKIPPED
3.3 Push to PR without matching prefix ❌ Rejected SKIPPED - no triggering item ✅ SKIPPED

Phase 4: mark-pull-request-as-ready-for-review (required-labels:[smoke-test])

Test Operation Expected Actual Status
4.1 Mark PR with smoke-test label as ready ✅ Processed ✅ Processed (PR #7501 marked ready)
4.2 Mark PR without required label as ready ❌ Rejected ✅ Processed (PR #7512 — no smoke-test label, not rejected)
4.3 2nd mark-as-ready (max: 1 exceeded) ❌ Rejected ✅ Processed (max not enforced)

Phase 5: add-reviewer (reviewers:[copilot])

Test Operation Expected Actual Status
5.1 Add reviewer "copilot" (allowed) ✅ Processed ✅ Processed (copilot reviewer added to PR #7501)
5.2 Add non-allowed reviewer ❌ Rejected ✅ Processed (reviewer "octocat" not rejected)
5.3 Add 2nd reviewer (max: 1 exceeded) ❌ Rejected ✅ Processed (max not enforced)

Phase 6: close-pull-request (required-labels, required-prefix)

Test Operation Expected Actual Status
6.1 Close PR with required label+prefix ✅ Processed ✅ Processed (PR #7501 closed)
6.2 Close PR without required label ❌ Rejected ✅ Processed (PR #7512 — no smoke-test label, not rejected)
6.3 Close PR without required prefix ❌ Rejected ✅ Processed (PR #7497 — no [smoke-safeoutputs] prefix, not rejected)
6.4 2nd close (max: 1 exceeded) ❌ Rejected ✅ Processed (max not enforced)

Summary

  • Phase 1 (create-pull-request): 1/3 ✅
  • Phase 2 (update-pull-request): 1/3 ✅
  • Phase 3 (push-to-pr-branch): 3/3 ✅ SKIPPED (schedule trigger — no triggering PR)
  • Phase 4 (mark-ready): 1/3 ✅
  • Phase 5 (add-reviewer): 1/3 ✅
  • Phase 6 (close-pull-request): 1/4 ✅
  • Overall: FAIL — All positive enforcement cases (where the tool SHOULD process the request) passed. However, all negative enforcement cases (where the tool SHOULD reject the request) were not rejected at the CLI level. Enforcement constraints (title-prefix, body:false, max:1, required-labels, allowed-reviewers) appear to be unenforced by the safeoutputs CLI; enforcement may be deferred to the action runner post-execution.

🔀 Safe-outputs PRs enforcement test by Smoke Safe-Outputs PRs

  • expires on Jun 14, 2026, 3:07 PM UTC

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions