cr research: local-gemma code-review DAG optimisation (cr-skill-workflow)#11
Open
byapparov wants to merge 85 commits into
Open
cr research: local-gemma code-review DAG optimisation (cr-skill-workflow)#11byapparov wants to merge 85 commits into
byapparov wants to merge 85 commits into
Conversation
Cross-experiment hypothesis/metrics/results registry (Google Sheet) built on the existing cr-loop harness. Reuses cr-loop's benchmark, answer-keys, and per-PR scorer; adds aggregate + sheet-sync connective tissue. cr-loop back-filled as experiment #1. MCP-driven Sheet writes; CR-only scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…types Backward-compatible: default path and CLI unchanged. Enables scoring the same findings against both answer-key versions in one process.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… per-PR scores Implements Task 3 of the code-review research framework. aggregate() reads all PR-*.findings.json from a directory, scores each via cr-loop's score(), and micro-averages TP/FP/FN across PRs (sum first, then compute P/R/F1) into one ResultsRow. Includes CLI with --findings-dir, --answer-key, --baseline-f1 flags. 3 unit tests pass; smoke test against Phase-E (10 PRs) returns f1=0.446, deltaF1=0.077.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Local gemma4:12b-96k via ollama/opencode reviews current aictrl_main files with vs without the aictrl KG MCP. Reuses kg-ab-test's config toggle, cr-loop's scorer + Phase-E prompt, and the code-review registry. Gold is a frozen KG-neutral answer-key (Claude oracle, MCP off + manual triage). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
14 bite-sized tasks: scaffold + pin, ollama/opencode configs, MCP tool-calling smoke gate, reviewer prompt + builder, tolerant findings parser (TDD), condition runner, e2e smoke, task selection, KG-off Claude oracle, triage + answer-key compiler (TDD), 3-rep sweep, score via the code-review framework aggregate, Google-Sheet registry sync, report. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(Task 8) Selected from aictrl_main @6b6bb39b via 3 parallel concern-area sweeps (service, API/MCP, data layer). Single files span scoring, crypto, auth-key races, pagination, route validation; modules chosen for cross-file coupling (validation pipeline, integration CRUD, finding dedup, KG content-fetch chain, RBAC). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two-pass, KG-off (aictrl MCP not connected to session): 20 oracle agents proposed 57 candidate findings; 20 independent adversarial verifiers re-checked each against the code and assigned verdict/action. Adversarial pass refuted task 101 (unreachable NaN), 203#4 (path-norm), and 4 of 5 RBAC claims in 205 (auth-disabled-mode unreachable). Result: 57 labels (49 TRUE / 4 FALSE / 4 UNCERTAIN) compiled to answer-key.json. Adds compile-answer-key.ts (+TDD). Candidates + triaged kept as provenance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… doc feasibility - opencode control path proven end-to-end: custom ollama provider (bundled openai-compatible) → gemma4:12b-cr @ 32k ctx, 100% GPU, ~60 tok/s. - 96k-context tag is impractical on 12GB VRAM (CPU spill); pinned 32k variant instead. Run with an empty scratch --dir (opencode snapshots its workdir). - gemma4 tool-calling validated natively (emits query_context-style tool_calls). - Treatment KG MCP blocked: aictrl.dev returns 401/OAuth; kg-ab-test bearer is dead. Needs a credential decision (fresh bearer / one-time OAuth / local KG). Feasibility written up in analysis/report.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tested 32k vs 64k: same 60 tok/s, same 8.1GB model size, 10.3GB total VRAM (1.4GB free). gemma sliding-window + flash-attn keep KV cache tiny. 64k gives treatment-context headroom; 96k still spills to CPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
opencode now connects to aictrl.dev/aictrl/mcp headless; gemma4 calls aictrl_query_context. MCP exposes 6 tools. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…risk #2 resolved) Both conditions validated headless. query_context returns real cross-file caller data for our task files. Note: restrict treatment agent to query_context (MCP also exposes record_* write tools). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ive ollama KEY FINDING: gemma4 thinking-mode swallows review output — via opencode (thinking forced on) a 205 review takes 127-236s and returns []; opencode ignores options.think:false. Native ollama /api/chat with think:false produces real findings in 11s and catches the gold privilege-escalation bug (205-C1). Decision: drive native ollama /api/chat (think:false) directly; opencode configs + review agent kept as reference only. Prompt, build-prompt, parser, scorer, registry unchanged. Tool restriction validated (treatment exposes only query_context; record_* blocked). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… approaches)
options.think:false, thinking:{type:disabled}, and the schema's reasoning:false
all leave thinking on. Root cause: opencode uses ollama's OpenAI-compat /v1
endpoint, which has no thinking param (ollama think is native-/api/chat only).
Native ollama runner confirmed as the path; seeds via native options.seed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run-native.ts drives ollama /api/chat with think:false. control = no tools; treatment = query_context tool loop against aictrl MCP (X-API-Key), record_* never exposed. Reuses buildPrompt + extractFindings. Per-rep seeds via native options.seed. Smoke: control 205 → 3 findings/6s, treatment 205 → 1 finding/5s (real findings, no more empty []). Open: gemma still under-calls query_context (0 on 205); LLM line-number drift will route some real catches to novels (cr-loop-style triage handles it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…um_predict 4000 think:true @ num_predict 16000 on 205: 267s, 15049 tokens, done_reason=stop, content=[] (thinking drafts findings then concludes nothing). think:false = 3 findings in 6s. Budget was not the cause. Runner stays think:false, num_predict bumped to 4000 for reason-in-content headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ma4 (thinking) 6 opencode routes + Modelfile param all fail to disable gemma4 thinking; raw /v1 reasoning_effort:none works. opencode's openai-compat provider doesn't forward reasoning controls to ollama. aictrl CLI (opencode fork v0.3.2) = same gap. Production runtime works only with GLM-5.1. Native runner required to test local gemma; relatability comes from porting the real skills (explore-context + fullstack-code-review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Case) Correction: earlier 'opencode cannot' conclusion was wrong — wrong param name. opencode openai-compatible schema only accepts camelCase reasoningEffort; maps to body reasoning_effort. snake_case reasoning_effort is silently dropped (overwritten undefined at openai-compatible-chat-language-model.ts:175). With reasoningEffort:none, 205 review = 4 findings in 14s. Production-relatable path (opencode/aictrl run) restored. Confirmed vs upstream issues (opencode#21903, openclaw#13575/#33272, ollama#12004/#14820). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… skills
aictrl run (the prod code-review CLI) + ollama gemma (reasoningEffort:none) +
aictrl KG MCP (X-API-Key) + the real explore-context/code-review skills
(skills.paths -> runtime-skills/). On module 205: MCP connects (toolCount=6),
skills load, and gemma organically calls query_context 4x (vs 0 in the native
runner with skill-as-prompt-text), flagging line 412 on the gold bug. The
explore-context skill via the real skill system is the KG-usage lever.
Adds aictrl-{treatment,control}.jsonc + runtime-skills/{explore-context,code-review}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…CP disabled run-aictrl.sh drives aictrl run per task/condition/rep, parses findings (parse-session.ts), and persists per-review timing + query_context counts (results/timing.csv + PR-*.meta.json). Control config now sets mcp.aictrl.enabled=false — AICTRL_CONFIG merges with the user's global config (which defines the KG MCP), so omitting it wasn't enough; verified control now makes 0 query_context calls vs treatment's 4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- aictrl run </dev/null: it was consuming the while-loop's piped task list, so each invocation ran only the first task (6/120 instead of 120). - KG count: 'grep -c ... || echo 0' emitted '0\n0' on no-match (grep -c prints 0 AND exits 1), corrupting timing.csv rows + zero-KG meta.json. Now captured as a single clean integer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unner The pinned git worktree (../aictrl_main-cr-pin) was pruned mid-sweep, so file reads failed at task 112 and set -e cascaded (only 11/120 ran). Fixes: - snapshot-task-files.sh: copy the 20 tasks' files from pinCommit into a stable local task-files/ (gitignored); pinDir now points there, not the worktree. - run-aictrl.sh: array-based loop (no stdin-eating pipe), per-task file-exists skip, fail-soft parsing, drop set -e so one bad task can't kill the sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/timing)
Reuses cr-loop score() over results/raw/rep-*/cond/{files,modules}; reports
P/R/F1 per condition and per task-class, plus KG-usage + time/review from
timing.csv. Rep-1 preview: control F1 0.049 vs treatment 0.182 (KG used 15/20).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aggregate() over results/raw + answer-key + session.json → micro-avg P/R/F1 by condition/class, ΔF1, KG-usage + action histogram, OUTPUT tokens/steps, time. 6/6 unit tests (math, token-summing, KG-action parse, class split, fail-soft). Corrects earlier ad-hoc over-counts: tokens summed only from step_finish events (message_complete duplicates ~3x); KG calls from session tool_use (log permission= over-counts). CLI: pipeline.ts --reps 1,2,3 --out results/summary.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
timing.csv: 120 rows complete (control 60 + treatment 60) summary.json: micro-avg F1 by condition+class, KG usage, tokens, time Key results: control F1=0.057 (file=0.029, module=0.092), P=1.0, 961 tokens/review, 30s avg treatment F1=0.103 (file=0.112, module=0.092), P=1.0, 290 tokens/review, 19s avg ΔF1=+0.046 overall; KG used 71.7% of treatment reviews, avg 1.5 calls Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…5 defs - FINDINGS.md: full writeup. Winner exp-003 5-specialist union (FULL per-run F1 0.439 / union-3 0.493, ≫ 0.355 goal). Real-set ~0.30, precision-bound. - exp-004 (strict judge) + exp-005 (soft judge): negative-result definitions kept for provenance. - Static analysis assessed as not tractable on isolated snippets (semantic bugs); KG-prefetch script node is the recommended next lever. - Results + hypotheses (H-008/009/010) synced to Google Sheet.
… context)
- kg-query.ts: direct MCP client for query_context (no model in loop).
- kg-prefetch.ts: script node — per-file impact + co_changes + per-function
caller counts → compact context block. No AI-node budget.
- run-workflow.sh: harness now supports type:script nodes (emit a context
artefact exposed as {{node.context}}, no findings).
- exp-006: KG-prefetch feeds all 5 specialists (union). Probe: precision nudged
up (FP 24→21), recall down slightly; full sweep running for the real number.
… (deferred) H-011 (running): each finding must carry repro steps + a failing unit test; drop unsubstantiatable ones. Generation-time precision gate vs the failed judge node. H-012 (deferred): EXECUTE the tests on the spot for ground-truth FP removal — blocked on runtime (needs buildable/runnable repo, same as static analysis). exp-006 KG-prefetch abandoned mid-sweep (marginal + slow) for this higher-EV lever.
For code review a missed real bug usually costs more than a false positive, so F2 is the decision metric. F2 flips the optimal strategy: under F1 union hurts the real set, but under F2 exp-003 union-of-3 is best (real F2=0.425). Printed for full+real, per-run+union; included in scores.json via prf().
exp-007 proof-obligation final (3 reps): real F1 0.211 / F2 0.181, precision 0.29, novels 103→25. Cuts FP+hallucination but recall collapses → worse F1/F2 than recall-max, best precision. F2 (recall-weighted) flips optimal to union-of-3 (exp-003 real F2 0.425 / full F2 0.571). Conclusion: maximise recall for F1/F2; proof-obligation only wins if the goal is precision. Sheet updated (F2 column, E-006/E-007 rows, H-011 outcome).
Doubles AI-node budget to 10 fine-grained parallel lenses (authz/injection/ secrets/nullsafety/logic/concurrency/resources/validation/errors/boundary), unioned. Probe (real-bug-rich, rep-1): REAL F1 0.379→0.403, F2 0.433→0.462, recall 0.478→0.511 — TP up with FP flat (more real bugs, no extra FP). Full 3-rep sweep running. Terminal stays union (judge transition is a dead end).
Budget scale-up to 15 AI nodes, two topologies (both union terminal): - exp-009: 15 distinct fine-grained lenses → union (does breadth keep scaling past 10?). 10 exp-008 lenses + arithmetic/statemachine/apicontract/config/idempotency. - exp-010: 5 specialists ×3 instances → union (in-process self-consistency — capture the union-of-3-reps F2 gain via resampling, not more distinct lenses). Probing 5→10→15 progression on the real-bug-rich subset before any full sweep.
5/10/15-node probe series (real-bug-rich subset): distinct-lens breadth SATURATES at ~5 (10 and 15 distinct lenses flat at REAL F2 ~0.41-0.43). But 5 lenses ×3 instances unioned (exp-010) jumps REAL F2 to 0.586 / recall 0.79 — and beats exp-003 union-of-3-reps at equal FP+budget. Lever = resample proven lenses, not add distinct ones. exp-010 full 3-rep sweep running for the headline.
5 lenses x 3 rounds, carry-forward: rounds 2-3 see prior findings and focus on
uncovered code. Same 15-node budget as exp-010 (independent x3); isolates
directed vs independent resampling. Built with explicit {{node_rN.findings}}
template refs (no harness change). Queued behind exp-010 sweep (one GPU).
fix-forward variant deferred (mutating code breaks line-keyed oracle).
…13 confirmed Head-to-head (real-bug subset): directed rounds beat independent x3 — REAL recall 0.667->0.816 (20/24 real bugs), F2 0.523->0.581, FULL F2 0.566->0.685. Carry-forward 'focus elsewhere' converts overlap into coverage. Cost: more FP/novels (precision flat). Strongest config in study; full 3-rep sweep running. Sheet H-013 updated.
Single-pass review vs full oracle. Opus REAL F1 0.730/F2 0.665 (precision circular - it authored the oracle). Haiku 4.5 (honest, run as session subagents) REAL F1 0.354/F2 0.333, P 0.39, recall 0.32 (13/43), only 12 novels. Findings: one pass catches ~1/3 real bugs (hard recall ceiling); model strength = precision/cleanliness; cheap gemma ensemble matches Haiku recall at $0 (directed-resampling far exceeds it). Implied best pipeline: gemma high-recall generator -> strong-model/test verifier. Synced E-OPUS/E-HAIKU to sheet.
- analysis/nodes-vs-f2.png + plot script: REAL-F2 vs gemma calls, labeled by exp. Shows topology dominates node count (15-node F2 spans 0.42->0.58). - exp-012 (directed R=2, 10 nodes), exp-013 (3-lens union), exp-014 (single review) to fill the directed-family saturation curve (N=1,3,5,10,15). Scale batch running.
…onse curves Self-contained HTML: KPIs, headline results, 7 inline-SVG DAG-shape diagrams (single/union/judge/proof/independent/directed/KG-prefetch), and two Chart.js response curves (LLM calls -> F2, total tokens -> F2) with off-curve topology variants showing shape dominates size. Recommends directed-resampling panel; N=10 efficiency knee.
- DAG diagrams rewritten in robust Mermaid (explicit edges, class statements per line) — fixes the v11 syntax error. - Charts: (1) calls->F2 quality curve with single-pass model stars (gemma/Haiku/ Opus; GLM pending); (2) nodes->tokens per review; (3) nodes->cost per review (Vertex AI Flash-class rate assumption). Keeps topology (node count) as the focus.
GLM 4.7 single-pass on real-bug subset: REAL F2 0.280 (recall 0.25, precision 0.55) — tied with Haiku 4.5 (0.29), above gemma (0.19), below Opus (0.57). NOTE: GLM was run via the Z.AI Coding Plan subscription (zai-coding-plan/glm-4.7); the standard zhipuai endpoint hung (7+ min/task).
- Proof-obligation diagram: 5 parallel lens nodes (consistent with union), not one box. - Directed diagram: rounds in a vertical subgraph -> cleaner alignment, no crossing edges. - Graphs 2&3: add single-pass cloud model ★ (Haiku 4.5 3.6k tok/$0.0052; GLM 4.7 5.6k tok/$0.0067 token-rate, Coding Plan ~$0). Note gemma per-call tokens ~4x the single-pass models because it loads the explore-context skill every call.
…/review vs F2) gemma DAG vs single-pass Haiku 4.5 / GLM 4.7 / Opus 4.8, REAL F2 vs $/review (log). gemma DAG is Pareto-dominant: @$0.007 (N=3) beats GLM/Haiku at equal cost; @$0.036 (N=15) matches Opus F2 (0.58 vs 0.57) at ~1/3 Opus's ~$0.10/review (local=$0). Opus precision circular. Rates: Haiku $1/$5, GLM $0.6/$2.2, Opus $15/$75 per 1M.
Force clean log ticks ($0.002/0.005/0.01/0.02/0.05/0.10) instead of colliding auto-ticks; softer gridlines; fixed min/max with padding for the Opus point.
… + exp-016 thinking - run-workflow.sh: experiments/<id>/aictrl.jsonc overrides the shared config (lets an experiment flip thinking on, etc.). - exp-015 (H-014 proof-core directed): proof-obligated round 1 -> directed rounds. Probe: REAL F2 0.510 / P 0.262 vs exp-011 0.460 / P 0.224 — precision edge at equal recall (within variance; promising). - exp-016 (H-015 thinking): gemma4:12b-cr has native thinking capability; enabled via aictrl.jsonc. ~5x slower (74s/call). Probe running. - GLM config (zai-coding-plan) committed.
Diversify passes by persona/stress framing, not topic or extra nodes (GPU time is the constraint). 5 general-review nodes: on-call SRE / mentor / attacker / forensic auditor / 3am-incident. Tests whether affective framing activates different model behaviour -> diverse findings, at the same 5-node budget as the topic-lens union. Queued behind exp-016 (thinking).
… resampling Add the first LOCAL precision signal for gemma code review: per-finding consensus voting across decorrelated resamples (votes = # resamples agreeing), calibrated for F2 on held-out tasks — not a model judge (those delete recall). - output: vote merge in run-workflow.sh (cluster file+line±5, keep all, annotate votes + nLenses; recall preserved) - score.ts --min-votes gate (agreement, not uncalibrated self-confidence) - rank-f2.ts: train/test-split F2 calibration of the vote cut (overfit guard) - vote-analyze.ts: retroactive vote analysis from persisted per-node files (0 GPU) - coverage-map.ts: structured coverage-map script node (already_found / unexplored_symbols / unexplored_failure_modes) replacing free-text "look elsewhere" - exp-020-coverage-directed (recall generator), exp-021-consensus-vote (precision) Offline result on exp-010's 15-voter data: REAL precision rises monotonically with votes (0.36 keep-all → 0.50 @≥3 → 0.80 @≥5 → 1.00 @≥6). exp-007 (directed) shows no signal, confirming consensus needs independent voters. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full-20 + probe results across the converged experiments: - thinking (exp-016b vs exp-014): discovery-killer, perfect confirmer (REAL P=1.0 R=0.04) — only viable as a verification node, not a pass - temperature: hot 1.1 (exp-018) best probe REAL F2 0.526 (R 0.667) — cheap recall lever - consensus vote (exp-021, full 20): HONEST CORRECTION — probe-only precision 0.80 did NOT generalise; held-out rank-f2 lift = 0.000. Vote-count is a real triage/ranking signal but not an F2 lever (F2's recall weighting punishes thresholding) - coverage-directed (exp-020, full 20): recall champion, REAL R 0.683 / FULL F2 0.607 - add exp-016b-think-single (clean thinking A/B) and exp-022-coverage-hot (coverage-directed @ temp 1.1, stacking the two recall wins) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical: per-task scratch ($SCRATCH/task-<id>-rep-<rep>) was shared across experiments and never cleaned, so the union/vote merge globbed leftover findings-*.json from OTHER experiments — inflating recall + vote counts (exp-022 max 18 voters on a 15-node DAG; exp-025 votes:17 on a 3-node DAG). - run-workflow.sh: wipe TASK_SCRATCH before each task (contamination fix) - run-workflow.sh: script-node failure/empty-context now EXCLUDES the task (writes PR-<id>.failed.json) instead of silently running without the KG/coverage treatment - score.ts / extract-novels.ts / rank-f2.ts: overlap() returns false (not true) on a null/unparsable line, so a line-less or comma-list finding no longer matches any oracle entry in the file - run-research-loop.sh: stage aictrl.jsonc (+ skills) so candidates aren't evaluated under the wrong temp/thinking/MCP config - clean-remerge.ts: recover correct results from persisted per-node files (re-merge legit nodes only) without re-running Recovery via clean-remerge confirmed contamination inflated recall (exp-022 REAL F2 0.509->0.486, recall 0.795->0.683); conclusions unchanged (exp-022 still the winner). New experiments exp-024/025/026/029 added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codifies the experiment-delivery discipline learned this session so future sweeps avoid the same failures: pre-flight config checks (MCP/skills only where KG is used; production-KG = direct aictrl run, not the DAG), verifying the treatment actually ran (query_context call count, failure-exclusion), post-run sanity invariants (votes <= #nodes catches scratch contamination; clean-remerge recovery), and scoring/comparability discipline (F2 metric, same-task-set only, single-rep noise). Separate from the domains/ run-experiment skill. RUNBOOK.md reduced to a pointer (skill is the single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…018/019 defs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…-semantic experiments Paper dir papers/4-cr-dag-codereview/: methodology, experiment-design, literature-review (verified refs), vendor-comparison, and the engineering blog post (Codex-drafted, fact-checked, rigour-hardened). Findings folded into the write-up: - per-round framing decorrelation lifts recall (clean B->C: 0.63->0.70) — the one decorrelation axis beyond temperature that paid (exp-030) - coverage-direction, tested properly after the template-var fix, is a non-lever (exp-031); fix applied to exp-020/022/023 prompts - a lens aimed at the hard classes (authz/dataflow/cross-fn) does NOT crack them (0/3 on its authz target, exp-032) — prompting isn't the lever - the floor: no never-found bug (reach 100%); ceiling is reliability + a fragile authz/cross-fn/dataflow tail reachable only by execution/cross-file - static linting (semgrep) ~0 yield here; noise is semantic Analysis tooling: never-found, rarely-found, recurring-fp, deepsem-vs-rare. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Research framework + experiments to maximise a local gemma4:12b-cr model's code-review F2 on a 20-task TypeScript benchmark, by iterating over DAG workflow shapes. Includes the
cr-local-kgbaseline study and thecr-skill-workflowDAG research.Headline results (REAL-set, recall-weighted F2 = decision metric)
aictrl run, code-review skill) is precision-heavy/recall-poor (3-5/40 real bugs); the multi-node DAG finds 5-9x more.Infrastructure
run-workflow.shDAG harness (model + script nodes; union/vote merges)score.tsdual full/real-set F1+F2 scorer;vote-analyze.ts,rank-f2.ts,clean-remerge.tscoverage-map.ts/kg-prefetch.tsdeterministic script nodescr-experiment-runnerskill — operator delivery runbookNotable fixes (this branch)
clean-remerge(conclusions held).Known caveats
answer-key.jsonhas 2 duplicate entries (oracle merge not idempotent) — flagged, not yet deduped.🤖 Generated with Claude Code