feat(eval): Move 1 — trajectory capture + TraceProbe INSIGHT scoring on real SWE-bench tasks#289
Open
theagenticguy wants to merge 5 commits into
Open
feat(eval): Move 1 — trajectory capture + TraceProbe INSIGHT scoring on real SWE-bench tasks#289theagenticguy wants to merge 5 commits into
theagenticguy wants to merge 5 commits into
Conversation
…on real SWE-bench tasks
Extends the variance-probe (@opencodehub/eval, spec 010) to score each run's
tool-call trajectory against TraceProbe's (arXiv:2607.06184) structural
anti-pattern detectors, and adds a SWE-bench Verified/Pro task generator so the
probe runs on real, graded tasks.
Roadmap Move 1 ("publish the search-loop reduction number"): the code-graph is
commodity; the moat only pays if the agent consumes it with fewer wasted steps.
This turns that into a measurable per-detector without−with delta.
- Capture: Claude Code switched to --output-format stream-json --verbose; Codex
--json already streams. New RunOutcome.trajectory.
- Normalize (trajectory.ts): both harnesses → TraceProbe's nine-type action
taxonomy; shell-wrapper unwrap for Codex's /bin/zsh -lc form.
- Detect (insight.ts): four STRUCTURAL no-oracle detectors — Search Loop,
Re-read Churn, Redundant Search, Shell-over-Tool. The four semantic detectors
(need an LLM labeler = judge-oracle caveat) are deferred to v2.
- Report: per-arm ArmInsight + per-harness insightDelta; --insight CLI flag.
- Real tasks (swebench.ts + scripts): instance → assertion task applying
test_patch and running FAIL_TO_PASS/PASS_TO_PASS, so correctness is graded
(upgrades Finding 0001's eyeball caveat). TraceProbe ran on SWE-bench Verified,
so per-detector numbers are comparable to the paper.
Fidelity (documented in Finding 0002): v1 grades on a /tmp clone without per-run
checkout isolation — token + trajectory deltas are the trustworthy headline,
assertion pass-rate is indicative; official Docker images are a v2 upgrade.
Off by default: report shape is unchanged unless --insight is passed.
118 eval tests + 335 cli tests green; pnpm check passes.
Live Bedrock measurement run and external publication of Finding 0002 are gated
on Laith.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…al-redos) CodeQL flagged 2 high-severity polynomial-backtracking regexes in the Move 1 code: - trajectory.ts SHELL_WRAPPER (`^\s*…sh\s+-…c\s+(.*)$`) — ambiguous \s* / \s+ backtracks on 'sh\t-c\t…' with many tab repetitions. Replaced with a linear tokenized unwrap (indexOfWhitespace / trimStart / isDashCFlag char scans); no regex over the command string. - swebench.ts trailing-slash strip (`/\/+$/`) — backtracks on an all-slash path. Replaced with a linear stripTrailingSlashes scan. Behavior preserved (existing shellFirstWord / instanceToTask tests green) plus regression tests for the tab-separated wrapper, a non-shell `-c` arg (gcc -c), and an all-slash cloneRoot. 121 eval tests pass; pnpm check green.
…MAX_ARG_STRLEN) The live N=2 pilot on SWE-bench flask exposed a real bug: every WITH-pack run errored (0 tokens, no trajectory), while without-pack ran fine. Root cause: the runner passed the composed prompt as a single argv argument, and an OCH pack context is ~1 MB. A single argv arg is capped at MAX_ARG_STRLEN (128 KB on Linux) — independent of the larger total ARG_MAX — so the with-pack spawn failed with "argument list too long" (E2BIG), silently, on every real repo. Fix: feed the prompt on stdin. `claude -p` with no positional reads the prompt from stdin; `codex exec -` reads it from stdin. argv stays tiny regardless of pack size. The runner already had an unused `stdin` seam — now it carries the prompt. After the fix, both arms populate on flask (with-pack 7.8M cache tokens vs without 0.6M — the pack IS in context). Tests updated: buildArgv drops the prompt positional; the injection test asserts the pack is on stdin and NOT on argv. 121 eval tests + pnpm check green.
…-reads) First live Move-1 run: 2 SWE-bench Verified tasks x 5 runs/arm, Claude Code / Sonnet 5 on Bedrock. Both tasks agree in direction — the pack suppressed every anti-pattern that fired and introduced none: - Shell-over-Tool 1.8 -> 0.3 firings/run (mean delta +1.5) - Re-read Churn 0.6 -> 0.1 (mean delta +0.5) - Search Loop 0 -> 0 — never fired; SWE-bench fix tasks don't trigger the >=10-action open-ended search stretches it detects. The honest, narrower claim is that the pack cuts shell-hunting and re-reads, not loops. Behavioral complement to Finding 0001's token cut. Codex arm deferred (this Bedrock account has only gpt-oss-*, not gpt-5.5). Assertion pass-rate 0/0 — deps not installed in the /tmp clones, so graded correctness needs the Docker path (v2); token + trajectory deltas are the trustworthy headline. Status Preliminary; wider sweep in flight.
…flow Sweep completed. 4 of 6 tasks gave a valid two-arm comparison; all 4 show the pack cutting shell-over-tool (mean 1.9 -> 0.65/run) and re-read churn (0.3 -> 0.05), Search Loop still 0 (fix tasks don't trigger it). Second finding from the 2 that failed: the pytest pack assembles to ~5.2 MB (~1.3M tokens), past Claude's 1M window, so every with-pack run errored with "prompt is too long". The default --budget bounds only AST chunks; xrefs / skeleton / file-tree are unbounded and dominate a large repo. Added as the top Next item: budget the whole assembled pack + have the probe skip an oversized with-pack arm rather than erroring all N runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Roadmap Move 1 — publish the search-loop reduction number. The code-graph is commodity now; OCH's moat only pays if the agent consumes the pack with fewer wasted steps. This extends the variance-probe (
@opencodehub/eval, spec 010) to score each run's tool-call trajectory against TraceProbe's (arXiv:2607.06184) structural anti-pattern detectors, and adds a SWE-bench Verified/Pro task generator so it runs on real, graded tasks.Finding 0001 gave the resource number (pack cuts tokens 2–4×). This gives the behavioral one underneath it: does the pack cut search loops — TraceProbe's most stable failure-associated clue?
The build (all additive;
--insightis opt-in, report shape unchanged without it)--output-format stream-json --verbose; Codex--jsonalready streams. NewRunOutcome.trajectory.trajectory.ts) — both harnesses → TraceProbe's nine-type action taxonomy; unwraps Codex's/bin/zsh -lc '…'wrapper so shell greps classify.insight.ts) — four structural, no-oracle detectors: Search Loop, Re-read Churn, Redundant Search, Shell-over-Tool. Predicates quoted from the paper's frozen-detector table.ArmInsight+ per-harnessinsightDelta(per-runwithout − with; positive = pack suppressed the anti-pattern).--insightCLI flag.swebench.ts+scripts/) — instance →assertiontask that appliestest_patchand runsFAIL_TO_PASS/PASS_TO_PASS, so correctness is graded (upgrades Finding 0001's eyeball caveat). TraceProbe ran on SWE-bench Verified, so per-detector numbers are comparable to the paper.Deliberate scope
Fidelity limits (documented in
docs/findings/0002)v1 grades on a
/tmpclone with deps installed; the probe runner does not reset the checkout between the N runs, so the token + trajectory deltas are the trustworthy headline and the assertion pass-rate is indicative. Per-run isolation / SWE-bench's official Docker images are a v2 upgrade.Tests
pnpm checkgreen: 118 eval tests (new:insight,trajectory,swebench; updatedcli-runnerfor stream-json), 335 cli tests, lint, typecheck, banned-strings.Grounding
CLI event schemas (Claude Code 2.1.x stream-json, Codex 0.143.x
exec --json) were live-captured, not recalled. TraceProbe's taxonomy + frozen predicates were pulled from the paper.Gated on maintainer
docs/findings/0002results table, flips status Preliminary).🤖 Generated with Claude Code