review: the re-review mode dial (full | scoped | flip-gated | fast)#246
Conversation
…live cases Phase 1 of the live A/B eval plan (#232). The corpus gains an opt-in live block carrying what a real model run needs and the deterministic replay does not: - prContext (PR title/description/author/base; the description is untrusted author text, so an adversarial case can carry its payload there or in the diff), - a post-change file tree on disk next to the case, via a new <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json (a directory containing case.json is one case; its tree is never parsed as corpus JSON), and - labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line window, mechanism keyword alternates). Live runs choose their own finding ids, so ground truth matches on anchor window + mechanism rather than id. The loader enforces the invariants: the live tag and the live block imply each other, a live case needs a cleanly-parseable diff, spec paths must appear in changedFiles and the diff, and every non-removed changed file must exist in the tree. loadLiveCorpus() returns the subset. The live half lives in corpus/live.ts; loader.ts re-exports it as the single public surface. Ten cases are converted with hand-authored real diffs and trees: five smoke incidents, both clean cases, one adversarial injection whose payload is a code comment in the diff, one golden holdout, and one synthetic mutation. Their recorded line anchors are rewritten to the authored defect lines (natural files beat content padded to synthetic line numbers), which activates the provenance gate on these cases in the deterministic suite; all expectations hold unchanged.
…action and case staging Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1 corpus format (#233). agent-extract.ts turns review.md's '## agent:' sections into data (name, description, pinned model, prompt body). It takes the markdown as a string with no fs access, because the baseline arm of an A/B reads the merge-base review.md via git show. Parsing is strict; a malformed or model-less section throws listing every problem, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real review.md (21 agents) and pins the staging-root reference so a future rename fails a test instead of silently staging nothing. live-stage.ts materializes the production staging layout for one live-enabled corpus case: pr-context.json (review.md Step 1's shape, synthetic identity fields), full.diff / pr.diff / full-stripped.diff (all the case diff; corpus diffs carry no generated files and no pattern-triage pass runs), files.json + review-files.json with the hasPatch cross-check derived from the diff parse, provenance.json, routing.json from the deterministic router, an out/ directory, and the post-change checkout copied from the case tree. rewriteAgentPrompt swaps the production staging root for the staged context dir. Everything sits behind an injected-fs seam and is memfs-tested. Model dispatch (phase 2c) is deliberately absent; it needs the Agent SDK dependency decision and arrives separately.
… runner (phase 2c)
live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.
live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.
live-match.ts scores a live run against a case's labeled defect specs: a posted candidate satisfies a spec when its anchor agrees with the spec's path (and line window when both carry one) and any mechanism alternate matches the finding's failure_scenario or prose; each candidate satisfies at most one spec and vice versa. An injected fallback arbiter (hard-capped, same-file only, recorded as via: fallback for audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. computeLiveMetrics aggregates recall, verdict agreement, clean false-flag (including a clean case that blocks), and noise. live-ab.ts is the arm orchestrator and CLI: baseline review.md from git show <merge-base>, candidate from the working tree, both arms over the same live corpus with everything else (corpus, lib, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model seam. Each arm runs under half the --max-usd budget with sticky exhaustion: once spend plus the running per-case average crosses the cap, remaining cases are recorded skipped and the report still emits. Spec-level regressions are diffed only over cases both arms scored. Judge scoring reuses the pinned judge (quality aggregates only; judge-vs-ground-truth disagreement keys on recorded ids a live arm does not use); the fetch model moves to judge-live-model.ts and live-judge.ts now imports it. runner.ts gains an optional RunOptions.validation override so a live validator's output replaces the recorded block. Report-only except the standing rule: adversarial-injection failures on the candidate arm exit non-zero.
Review Eval A/B runs on every non-draft PR touching workflows/review/** (and on workflow_dispatch): both review.md arms over the live corpus via live-ab.ts, with the delta report posted as a sticky PR comment (hidden-marker upsert), appended to the job summary, and uploaded as an artifact even on partial or gate-failing runs. Per-PR scope is the smoke-tagged live subset; the full-eval label or dispatch input lifts it, skip-live-eval opts out, drafts wait until ready, the changeset-release branch is excluded (it matches the path filter via package.json but changes no behavior), secretless runs skip green so fork PRs never fail, and per-PR concurrency cancels superseded runs. The workflow name is distinct from every gh-aw workflow per the operational-floor rule about shared concurrency groups. live-ab.ts gains --smoke-only and writes out/live-ab-report.md alongside the JSON for the comment step.
… and vitest Tree directories are byte-exact case fixtures paired with each case's diff: prettier auto-formatting one would silently desync it from the diff the provenance gate parses, and a tree may carry a *.test.ts whose tests fail by design (test-adequacy cases), so vitest must not execute them either. CI's lint job caught the first drift (prettier wanting to rewrap a fixture ternary).
…rpus' into jwbron/live-eval-producer-staging
…staging' into jwbron/live-eval-ab-runner
…to jwbron/live-eval-ab-ci
…s with the case id Live agents choose their own finding ids, so every case's first correctness finding was live-correctness-reviewer-1; ids were unique within a case but collided across cases, and judge.ts's score join requires arm-global uniqueness. Caught by the first real A/B run (both arms completed, then judge aggregation threw). Ids are now <caseId>:<id> from the moment of parsing, so claims.json, the validator round-trip, the matcher, and the judge all see the same namespaced id.
…staging' into jwbron/live-eval-ab-runner
…eport instead of killing it The first real A/B run spent both arms' budgets and then died in judge aggregation, writing no report: the exact everything-spent-nothing-posted failure mode the plan forbids. Judge scoring is additive, so a per-arm failure is now caught, recorded as judgeError on the arm, rendered as a degradation note in the report, and the run proceeds to write JSON + markdown and evaluate the adversarial gate as usual.
…to jwbron/live-eval-ab-ci
…A/B phase 5) Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as a Claude Code skill so a trial costs an operator an afternoon instead of a week of hand choreography. The skill collects required inputs (seeded branch, human-authored defect table, arms, budget approval) and refuses to improvise ground truth; sets up one isolated PR per arm with per-arm trigger recipes and the distinct-workflow-name rule (same-named gh-aw workflows share a per-PR concurrency group and cancel each other); collects reviews, artifacts, and costs per run with the known gh-aw artifact-bug tolerance; drives optional lifecycle pushes; scores defect by defect with the deterministic rule mirroring eval/live-match.ts plus audited manual judgment; exports live-enabled corpus case skeletons with a sanitization gate for private-repo content landing in this public repo; and cleans up trial PRs, branches, and temporary workflows. Trials remain the architecture-bet instrument; per-change evals belong to the corpus A/B. .gitignore narrows from .claude to .claude/* with a !.claude/skills/ carve-out: local agent state (settings, worktrees) stays untracked, project skills are shared tooling and are committed.
…orpus' into tmp-refresh
…roducer-staging' into tmp-refresh
…b-runner' into tmp-refresh
…b-ci' into tmp-refresh
…ity, code-rendered from the reconciler keep-list
… instead of dropping them
…oad (allowed-paths must match staging-relative paths)
🦋 Changeset detectedLatest commit: 9b047d5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
jwbron
force-pushed
the
jwbron/rereview-mode-dial
branch
from
be4202d to
6cee4a1
Compare
jwbron
changed the base branch from
jwbron/review-out-of-lane
to
jwbron/review-dispatch-tax
khan-actions-bot
requested review from
a team and
jeresig
and removed request for
a team
… into jwbron/review-dispatch-tax
… into jwbron/rereview-mode-dial
…ereview-accountability
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| input.currentSignature, | ||
| input.priorStamp.anchorHunks, | ||
| ); | ||
| if (divergence.unreviewedShare >= threshold) { |
There was a problem hiding this comment.
suggestion (non-blocking): The inclusive >= boundary is the whole point of the tripwire, but share === 0.4 exactly is never tested — the decideReReviewDepth cases cover 1.0/0.5 above and 1/6, 1/3 below the wire. A refactor to > would slip a diff sitting precisely on the threshold through as a cheap re-review with no test failing. Add a case with 2 of 5 unreviewed hunks (share 0.4) asserting depth === 'full' and tripwireRearmed === true.
Low-confidence (1)
workflows/review/lib/counters.ts:291—costByRereviewDepth: a depth group mixing cost-bearing and cost-less runs dividesusdSumby allruns, understating per-run cost; only uniform groups are tested (may be intentional, mirroringtokensPerRun).
| * `full` is the default everywhere: a repo pays for a cheaper mode only by | ||
| * writing a `re-review` line in its ROUTING file. | ||
| */ | ||
| export const RE_REVIEW_MODES = [ |
There was a problem hiding this comment.
thought (non-blocking): The changeset's own evidence ("a fresh seeded defect was caught on a re-review push that a reconcile-only path would have missed; the case for scoped as the recommended reduced depth") argues for full|scoped, yet fast and flip-gated — both reconcile-only for fresh code — ship now as permanent surface before any consumer has priced even scoped. Consider shipping full|scoped and adding the reconcile-only modes in a follow-up if the live A/B shows scoped doesn't capture enough of the savings. (Counter-argument acknowledged: the A/B needs the modes to exist to price them.)
| * pushed onto a sparse PR (share near 1.0) always re-arms. Exported so the | ||
| * eval suite and the live A/B can price other settings. | ||
| */ | ||
| export const DEFAULT_TRIPWIRE_THRESHOLD = 0.4; |
There was a problem hiding this comment.
thought (non-blocking): computeDivergence uses an unweighted hunk-count share — no weighting by changed lines or ROUTING risk tier — so under fast a single hunk pushed onto a 6-hunk reviewed PR sits at ~0.17 and never trips the 0.4 wire, getting zero finding-producing review. Both adversarial lifecycle cases use payloads that dominate the count, so this small-payload weak spot isn't exercised. A floor rule (any unreviewed hunk in a high-tier path re-arms) or a line-weighted share would close it. Not a regression today since full is the default.
| a category call. Count the blocking labels in your final comment set; zero blocking | ||
| labels means APPROVE. | ||
|
|
||
| **The re-review flip rule (reduced depths only).** One addition to the rule above |
There was a problem hiding this comment.
question (non-blocking): The flip gate is computed as data (flipGate in the plan, keptBlockingCount in rereview.json), but I don't see any code comparing the submitted verdict against it — computeVerdict keys only off posted labels, and the veto lives entirely in this Step 4 prose. Is a code chokepoint (e.g. the stamp CLI refusing an APPROVE stamp when depth is flip-gated/fast, the prior stamp verdict was REQUEST_CHANGES, and keptBlockingCount > 0) intended as a follow-up, or is prompt-following the deliberate enforcement for this guard?
# Conflicts: # workflows/review/eval/corpus/live.ts
| * world (its cwd, with no network), so the agent WILL often try to read | ||
| * an imported module or caller that is not there; that read returns an | ||
| * ordinary not-found tool error the agent tolerates and works around, | ||
| * not a run failure. The cost of a missing file is realism, not a crash: |
There was a problem hiding this comment.
thought (non-blocking): The "not a run failure" guarantee for a missing tolerated read is enforced elsewhere (live-runner.ts absorbs per-tool read errors and throws only on run-level subtype !== "success"), and no test near this comment pins it. The corpus-side promise can silently drift from the harness if the staging/runner behavior later tightens. Consider cross-referencing the runner or adding a small assertion so the two stay coupled.
jeresig
left a comment
jeresig
left a comment
There was a problem hiding this comment.
Great! Pushing back a bit so that we move more of this staging logic into a pre-workflow step, rather than relying on the agent to run this.
| RIGHT-side line numbers of the added lines inside in-scope hunks. Step 3 uses this to | ||
| filter candidate comments. | ||
|
|
||
| **Stage the bot's prior reviews.** Fetch the PR's reviews (`pull_requests` |
There was a problem hiding this comment.
Musing: I think we would save a lot of money/tokens/time by moving a lot of this (and similar staging) logic into Github action steps that run before the workflow that downloads and stages everything and then it can just read what it needs, if it needs it.
| **Decide the re-review depth (deterministic code).** After the provenance CLI, run | ||
| the re-review mode CLI from the shared lib checkout, once: | ||
| ``` | ||
| cd gh-aw-review-lib && npx -y tsx workflows/review/lib/rereview-mode.ts |
There was a problem hiding this comment.
If we did the pre-run we could run these scripts ourselves and then just feed in the results in here (which would also improve speed and reduce token usage).
| ``` | ||
| cd gh-aw-review-lib && npx -y tsx workflows/review/lib/rereview-mode.ts | ||
| ``` | ||
| It reads `routing.json` (the repo's `re-review` mode line, default `full`), |
There was a problem hiding this comment.
I think some of this context isn't needed, the reviewer here only cares about what the file is, not how it arrived at that solution.
| - **`depth: scoped`**: the full roster runs, but over only the unseen hunks. Before | ||
| Phase 1, overwrite `/tmp/gh-aw/review/full-stripped.diff` with the contents of | ||
| `scoped.diff`, and in Phase 1 build `pr.diff` from the `scoped.diff` sections of | ||
| the triage `reviewFiles` (a `reviewFiles` entry absent from `scoped.diff` is | ||
| already reviewed; leave it out of `pr.diff`). Everything else, the provenance |
There was a problem hiding this comment.
Again, this feels like stuff that we could just set up ahead of time so that it can't ever get it wrong.
|
Agreed, and this is where round three is headed: the deterministic-orchestrator work covers exactly this. The staging layer you flagged (the prior-reviews fetch, the rereview-mode CLI run, the scoped-diff swap, plus the existing provenance and new-scope steps that share the pattern) needs no model output, so it can all run as pre-workflow steps the agent just reads. Rather than special-casing this PR, I'll fold it into that migration so the whole layer moves at once and the eval producer runs the same pre-steps (otherwise the A/B stops measuring what production does). Merging this as-is since the dial defaults to |
…tring (review feedback)
Review Guidancegithub-actions (6 files)
Common patterns3 files: New 2 files: Re-exporting the new routing-config symbols ( 2 files: |
| * squashes, and base merges (which rewrite SHAs and shift line numbers but | ||
| * not the change's content), while a deletion-only payload still moves the | ||
| * fingerprint. (Step 1's older added-lines-only prompt definition is | ||
| * superseded by this CLI; the one-time mismatch on upgrade degrades to a |
There was a problem hiding this comment.
suggestion (non-blocking): The docstring calls Step 1's added-lines-only hunk signature "superseded by this CLI," but the two coexist with different rules: this CLI hashes added and removed lines, while review.md Step 1's comment-scoping signature is still added-lines-only and live, so they disagree on a deletion-only change. Consider unifying on one shared helper, or rewording to say the two signatures are intentionally distinct (comment-scoping vs. divergence).
Low-confidence (1)
workflows/review/lib/rereview-mode.ts:328—findLatestStamp/parseRereviewStamp: the stamp is unauthenticated and parsed last-match-wins from a body that can carry author-influenced text, so a foreigngithub-actions[bot]review echoing a stamp-shaped comment and newer than any genuinely-stamped review could anchor the run. Latent forfast/flip-gatedonly — defaultfullnever reads the stamp.
| /** Truncation keeps the stamp compact; 16 hex chars ≈ 64 bits per hunk. */ | ||
| const HUNK_HASH_CHARS = 16; | ||
|
|
||
| const HUNK_HEADER_RE = /^@@ -\d+(?:,\d+)? \+\d+(?:,\d+)? @@/; |
There was a problem hiding this comment.
note (non-blocking): HUNK_HEADER_RE and the hunk-body split are hand-rolled here, but diff.ts (already imported for splitUnifiedDiff) declares an identical HUNK_HEADER_RE (unexported) alongside countHunkHeaders/stripDiffFiles. Exporting a shared hunk-body splitter from diff.ts would keep the grammar in one place so a future diff-format tweak can't drift between the fingerprint and the rest of the pipeline.
…' into jwbron/rereview-mode-dial # Conflicts: # .claude/skills/review-trial/SKILL.md # workflows/review/lib/disciplines.test.ts # workflows/review/lib/rereview.test.ts # workflows/review/lib/rereview.ts # workflows/review/review.md
…under the max-lines limit (the main merge pushed router.ts to 1004)
jwbron
force-pushed
the
jwbron/rereview-mode-dial
branch
from
481fb40 to
9b047d5
Compare
|
Follow-up for the staging migration discussed above (repo issues are disabled, so recording it here; the round-three plan carries it as step 5's second named slice): Stage a deterministic caller mapThe gap, observed liveThe v1.5.0 behavior test (Khan/webapp#40793, seeded per the #40690 methodology) planted a cross-file amplification: a public function's Why this is structuralThe pipeline deterministically computes facts about the diff (changed-line provenance, snap tables, routing, budgets) but nothing about the diff's dependents. "Who relies on the contract this change narrowed?" is left to model judgment under an investigation budget deliberately designed to discourage speculative exploration, and the camouflage is systematic: a contract change that ships with an in-diff consumer answers the cheap "why does this exist" question without any caller ever being read. Exhortation ("investigate harder") fights the budget discipline; the fix is a missing deterministic input, the same class of gap Proposed fix (a named slice of the deterministic-orchestrator staging migration)
Deliberately out of scope: an escalation edge that lets a design-altitude observation re-dispatch the correctness reviewer (model-driven control flow exactly where the roadmap removes it), and dynamic-dispatch/cross-service call resolution (real but much smaller residual than today's gap). Refs: round-three plan step 5 (second named slice) and step 9 (first new mint); the behavior-test scorecard on webapp#40793; eval memo rev 5.1. |
2 participants
The re-review mode dial: the runs-per-PR cost lever from the plan of record's "Re-review economics" section. Stacked on fold-in batch two (#244 -> #245 -> #247, base
jwbron/review-dispatch-tax); the eval stack (#232-#238, #243) merged to main 2026-07-10.Why
Two measured lifecycles of the same seeded PR paid the full 11-agent fan-out on every push, with per-run cost rising as threads accumulate (v1.3.1: $7.19 / $8.77 / $9.14; v1.4.0: $8.31 / $9.87 / $11.29, runs 29044383251 / 29045581242 / 29046928431 in Khan/webapp). The approval run is the most expensive while emitting the least. Both lifecycles also caught a fresh seeded defect on a re-review push that a reconcile-only fast path would have missed, which argues for
scopedas the default reduced depth.What
A per-repo mode line in
.github/aw/review/ROUTING:full: today's behavior, and the default. No consumer changes behavior until its repo sets a mode line.scoped: full roster, staged only the hunks new since the last fully-reviewed fingerprint; comments scoped to those hunks.flip-gated: reconcile plus the correctness pass over new hunks; a REQUEST_CHANGES to APPROVE flip is vetoed by that pass's validated blocking findings (the findings gate the flip instead of being discarded).fast: reconcile-only.Three guards, all deterministic (
workflows/review/lib/rereview-mode.ts):Named interactions handled by construction:
state, so a dismissed approval still anchors and still reads as an approval.Eval and measurement
The roster-level evals live in #251 (stacked on this PR): open-PR corpus cases with reconciler dispatch and flip-gate/duplicate scoring, the per-mode pricing sweep (
eval/rereview-sweep.ts), and the ported seeded lifecycle. This PR carries the deterministic layer:eval/lifecycle/):rewrite-after-approvalandsparse-pr-then-payload, each scored as "the tripwire re-armed and the payload got a full review", plus an economy control (fix-push-stays-cheap) scored as "the reduced path actually ran". Replayed deterministically ineval/lifecycle.test.tsunder all three reduced modes.out/rereview-plan.json;lib/counters.tsaggregatescostByRereviewDepth; the review-trial skill gains a ROUTING-override arms pattern (two arms at the same ref differing only in the mode line) and scores adversarial lifecycle pushes.enablelines are.Composition with batch two
Based on the top of the batch-two stack (#243 -> #244 -> #245 -> #247) so the mode dial's re-review paths compose with the re-review accountability rendering (#244) rather than race it: the accountability section stays the visible body surface; the fingerprint stamp is appended last as a hidden comment; and the flip gate reads
keptBlockingCount, a small additive field this PR adds to the accountability renderer'srereview.json, so "did the reconciler keep any blocking thread" is a number read at verdict time, not a label judgment re-made there. The orchestrator wiring (Step 1 prior-review staging, the Step 3 depth plan and per-depth dispatch/staging, the Step 4 flip rule, the Step 6 stamp, reduced-depth skips in Steps 7/8) is the last commit on the branch.Supersedes
Khan/frontend#13420 (frontend's consumer-side re-review stopgap, held on the old pin) is superseded by this: the dial lives in the shared workflow and frontend opts in with a single
re-reviewline in its ROUTING file once priced. Frontend is deliberately not touched here.Follow-up: staging moves pre-workflow (2026-07-13)
Per review feedback (jeresig): the deterministic staging this PR wires through orchestrator prompt steps (the Step 1 prior-review fetch, the rereview-mode CLI invocation, the scoped-diff swap) needs no model output and will migrate into pre-workflow steps the agent just reads, folded into the round-three deterministic-orchestrator work rather than special-cased here. The eval producer moves with it so the A/B keeps measuring production. This PR's prompt wiring is interim and gets deleted wholesale when that migration lands.