Adopt structured marker contract for quality-gate (replace grep)

[cbeaulieu-gt]

Summary

Long-term direction for the claude-pr-review/quality-gate feature: replace the regex-grep approach with a structured contract between anthropics/claude-code-action (or a wrapper) and the gate. Grep is fragile by design and the marker-validation + multi-bot collision issues in this milestone are symptoms of that fragility — addressing them tactically buys time, but the root fix is a contract.

Why grep is the wrong long-term answer

• Prompt templates drift; regex misses become silent fail-open events
• Severity classification is currently encoded in human-readable prose, not data — every consumer that wants to act on it has to re-implement the parser
• Adding a new severity tier (e.g. "Note" or "Suggestion") requires every consumer to update their regex
• The grep can't reliably distinguish "Critical finding the reviewer raised" from "the word Critical appearing in the code being reviewed"

Proposal options

Option A — Hidden JSON block in the review comment

Emit a JSON block inside an HTML comment (invisible in rendered Markdown):

<!-- claude-pr-review-summary-v1
{
  "schemaVersion": 1,
  "findings": { "critical": 2, "high": 1, "medium": 4, "low": 0 },
  "verdict": "blocking"
}
-->

Gate parses with jq. Versioned schema; backwards-compatible drift.

Option B — Use anthropics/claude-code-action outputs

Consume structured findings as step outputs (or as an uploaded artifact). Requires upstream cooperation but is the cleanest integration.

Option C — Separate structured artifact

Have the action upload a claude-review-findings.json artifact; the gate downloads and parses it. Decouples the user-facing comment from the machine-readable signal.

Blocked by

• Fail-open issue — fix first; structured contract doesn't matter if the caller silently swallows failures
• Validate-regex-against-upstream — informs whether upstream cooperation (Option B) is viable
• Multi-bot selection — the sentinel from Option A or comment ID from Option B is the same mechanism that fixes selection ambiguity

Acceptance criteria

• Choose one of Options A/B/C based on what's feasible in anthropics/claude-code-action
• The gate consumes structured data, not grep on comment prose
• Schema is versioned (so prompt-template evolution doesn't break consumers)
• Test fixtures cover: schema-v1 → works; schema-v2 → works for v1 callers via compatibility shim; corrupted/missing → safe fail
• Documentation in pr-review/README.md explains the contract

Out of scope

• Choosing the severity taxonomy itself — orthogonal; this issue is about transport, not classification

Refs #176 #179

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

[cbeaulieu-gt]

Pausing the acceptance window — data collected so far is not representative.

Survey on 2026-05-09: the bar requires ≥20 reviews / ≥5 PRs / 0 shadow-gate disagreements / 0 marker_missing events. An unscoped 14-day sweep counted 71 runs with ~64 disagreements, but most of those predate PR #249 (the resolution-narration grep fix) and reflect the bug the fix addressed. Re-scoping to runs after #249 merged (2026-05-08T22:35:31Z) yields only 3 qualifying reviews — clean (0 disagreements, 0 marker_missing), but far below the ≥20 threshold.

Why we should not just wait for the count to climb: empirical verification under #245 (closed 2026-05-09, see verdict) confirmed that the baked agents/skills inside the runtime overlays are orphaned — not discoverable by the CLI at job runtime. The W3 synthesis agents are baked into the review overlay but never load. So the post-#249 runs we'd collect against the bar are running with a different agent set than the one we think we're testing — likely the consumer repo's .claude/ if any, otherwise nothing. Whatever signal we collect during this window does not measure the actual W3 path we want to cut over to.

Plan:

1. Hold the cutover. Acceptance checkboxes stay unchecked.
2. Wait for Make baked overlay tree (agents/skills/plugins/hooks) reachable from CLI at job runtime #259 to land — fixes both reachability and the ALLOWED_TOOLS allowlist that prevents Task/Skill invocation.
3. Reset the observation window after Make baked overlay tree (agents/skills/plugins/hooks) reachable from CLI at job runtime #259 ships. Counters restart from zero on the first run executing on a remediated overlay.
4. Resume normal cadence collection from there.

This issue is blocked by #259.

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt