diff --git a/scratch/reducing_agent_search_scope.md b/scratch/reducing_agent_search_scope.md deleted file mode 100644 index 6c41795..0000000 --- a/scratch/reducing_agent_search_scope.md +++ /dev/null @@ -1,491 +0,0 @@ -# Reducing agent search scope — methodology note - -*2026-07-04. Local draft. Not for the public repo.* - -## The problem, one line - -An agent driving a lab is paying tokens for the *entire universe of choice* on every discovery call. Every byte the tool returns is a byte the model must read on this turn and re-read on every subsequent turn (context accumulates). Reducing the search scope *at the tool interface* is the highest-leverage way to improve agent ability, precision, and cost — all three simultaneously. - -## Evidence — one controlled A/B, same goal, same model - -Two live-fire runs, same goal (*"how does DuckDB scale from small to large? DuckDB only, no Docker"*), same model (`anthropic/claude-sonnet-5`), same lab. Between runs, the only change was **taxonomy** on the discovery tools: a small `list_categories` vocabulary and a `list_suites(category=X)` filter, with the SQL payload gated behind `include_sql=true`. - -| Metric | Before | After | Change | -|---|---|---|---| -| Turn-1 tool result (bytes) | 88,480 | 1,715 | −51× | -| Turns to final answer | 12 | 8 | −33% | -| Total tokens (in + out) | 626,601 | 111,575 | **−5.6×** | -| Outcome | clean final analysis | clean final analysis | same quality | - -Same-quality analysis, one-fifth the cost. Ability and precision were not sacrificed; if anything, precision improved (the second-run agent fetched two candidate templates in parallel on turn 3, compared them, then submitted — a move it didn't make with the noisier context). - -## Why the effect is compounding, not one-shot - -Naive view: "we saved 87 KB on turn 1." Actual mechanism: **context accumulates linearly per turn**. Every turn's prompt includes all prior tool results. A single 88 KB payload on turn 1 costs 88 KB × N remaining turns worth of prompt tokens, not 88 KB. That's why a −51× reduction on one call produced a −5.6× reduction on the total. - -The saving multiplies over the length of the run. Fixing the biggest early payload is disproportionately valuable. - -## The mechanism — three moves the taxonomy makes - -1. **A small vocabulary the agent can survey.** 12 category names + one-line descriptions ≈ 600 B. The agent sees the shape of the choice space without seeing the contents. -2. **A filter argument on the noisy tool.** `list_suites(category=scaling)` returns the 2–3 suites tagged with `scaling`, not all 16. The filter is a *routing hint* the agent computed cheaply from step 1. -3. **Gated detail on the expensive field.** `include_sql=true` opts back into the raw SQL only when the agent decides it needs to reason about the SQL itself. Default response is names + tags + engines. - -The three moves compose. Each one alone would help; together they turn discovery into a series of small, targeted queries instead of one giant dump. - -## Where this generalizes (what to apply next) - -Any tool that currently returns *"everything the agent might want"* is a candidate: - -- **Fragments / raw results.** Already partially addressed by the granular projections (`get_experiment_summary`, `get_means_by_partition`, `get_scaling_factor`, `get_replication_stability`). Same pattern: small first, drill deeper by name. -- **Historical corpus** (when longitudinal comparison lands). Don't dump every past run of a config. `list_capsules(category=X, since=Y)` first; specific IDs second. -- **SQL content itself.** Even with `include_sql=true`, currently returns every benchmark's SQL for the suite. Could become `get_benchmark_sql(suite, benchmark)` — one at a time. -- **Tool listing itself.** When the tool inventory grows past ~20, the *tool schema* becomes another payload. Category the tools. Load only the category the agent's current task needs. -- **Skills.** Right now all skills are loaded upfront in the system prompt. Same pattern: `list_skills` (names + one-line summaries) → `get_skill(name)` on demand. Cheap. -- **AGENTS.md itself.** Currently loaded whole. Could be sectioned by task-type, with a lightweight index. - -Underlying principle: **progressive disclosure at the tool interface**. The agent must actively request more detail — never gets the whole universe on the first turn. Basic API design applied to LLM tool interfaces where every byte round-trips through the context window on every subsequent turn. - -## The precision claim is separate and worth naming - -The token-cost win is obvious. The *precision* claim is subtler: - -- With less noise in early turns, the second-run agent picked up `scaling_depth` as an alternative to `quickstart` and compared both templates — a move it didn't make in run 1. -- Deduplication worked cleanly: `submit_experiment` returned the same `803f3c94` as run 1 (identical config → identical content-addressed ID → the coordinator served the cached sealed results without re-executing). -- Fewer turns → shorter feedback loops → less opportunity for the model to drift (empty responses, wrong tool calls, or #7-style rigor-theater). - -*Less scope to search means more attention on the actual decision.* Not a novel finding in the literature — it's why humans use categories, filters, and dashboards — but worth stating as method for the lab's ruler: **the agent's tool interface is itself a research object**, and reducing search scope is a measurable intervention with a measurable effect (5.6× token reduction on this workload). - -## As a lab measurement, not just an engineering fix - -This is a data point the lab can produce on itself. Two runs, same inputs modulo the interface. The delta is attributable to the interface change. That's a legitimate methodology observation: - -- **Independent variable:** presence of category taxonomy + gated payload. -- **Dependent variables:** turn-1 payload size, total token cost, turns to convergence, final-answer quality. -- **Confound to control for:** model, goal, prompt version, temperature. Handled here — same run repeated with only the interface changed. - -Future runs across a model capability sweep (llama-3.1-70b, mixtral-8x22b, qwen-2.5-72b, deepseek-v3, sonnet-5, opus-4-6) can produce a *scope-reduction-effect curve*: does the win scale with model capability, or is it constant? Testable. - -## Open questions - -- Where's the floor? At some point the taxonomy is small enough that further hiding costs the agent more (extra turns to fetch what it needed anyway) than it saves. That trade-off is measurable — the JSONL trace captures per-turn tool selection, so we can see when the agent had to fetch multiple category slices vs. when one was enough. -- Does the effect hold on weaker models? SBD-2's llama3 hit workflow-capability failure with the old interface. Would the taxonomy have gotten it past turn 23? Unknown — worth re-running. -- What's the right unit of category? Suites is one level; the SQL benchmarks *within* a suite is another; the *questions* those benchmarks answer is a third. The methodology says "categorize", not "categorize at level X" — the level itself is a design choice per tool. -- Should the taxonomy be static (YAML) or derived (auto-tagged from suite metadata + capsule configs)? Static is cheap and controllable; derived stays in sync automatically but hides the vocabulary from the agent's introspection. - -## Extensions (Ramona, 2026-07-04) — three moves that follow from the same principle - -The taxonomy result is a small confirmation of a bigger claim: **the traces produced by these runs are themselves the data that answers the open questions the interface changes surface**. Interface change → traces → analysis → next interface change. The lab measures the agent measuring the lab, and each level uses the same discipline (sealed measurement, distilled decisions). It doesn't get more fractal than this. - -Three architectural moves that follow from the principle: - -### 1. Hierarchical taxonomy - -Flat 12 categories collapses distinctions the research actually needs (`analytical > aggregation > scaling` is not the same slice as `analytical > aggregation > selectivity`). - -- **Single-axis hierarchical** (parent/child in `taxonomy.yaml`): easy to implement, natural to navigate. Start here. -- **Multi-facet** (topic × complexity × engine-type as orthogonal axes): more expressive but higher navigation cost. Add when a specific research question forces it. -- Implementation shape: extend `taxonomy.yaml` with nested keys OR use `:`-delimited category names (`analytical:aggregation:scaling`) — the second is simpler to parse and doesn't require a schema change. - -### 2. Explicit agent state machine - -Current `autonomous_agent.py` runs an implicit state machine — every research read of the loop has to reconstruct where the agent was. Making it explicit unlocks three things: - -- **Per-state tool subsets.** In `discovery` state, only `list_categories / list_suites / get_template`. In `build`, only `submit_experiment`. In `analyze`, only the result-reading projections. Reduces the tool inventory the model reasons over on each turn — same principle as the taxonomy, applied to the tool list itself. -- **Per-state system prompts.** Discovery prompt frames a search problem; analysis prompt frames a synthesis problem. Different words for different tasks. -- **Machine-readable failure classification.** JSONL trace gains a `state` field per event. "Gave up in `analyze` at turn 20" is a different specimen from "gave up in `build` at turn 8" — currently both look identical in SBD-2's shape. - -States, first pass: `discover → build → submit → poll → analyze → answer`. Transitions gated by predicates (e.g., `build → submit` only after `submit_experiment` returns a valid `experiment_id`). - -### 3. Spawning sub-agents for granular tasks - -The biggest architectural fork. Instead of one generalist agent driving the whole loop, decompose into specialists: - -- `config_builder` — input: goal + category. Tools: `list_suites`, `list_templates`, `get_template`, `submit_experiment`. Output: `experiment_id` or a config-error. -- `poller` — input: `experiment_id`. Tools: `get_experiment_status`. Output: `complete` / `failed`. -- `analyzer` — input: `experiment_id`. Tools: the four projections + `get_experiment_result`. Output: structured claim + narrative. -- `orchestrator` — input: user goal. Delegates to the specialists; owns nothing but the plan. - -**Why this is more than a refactor.** - -1. Each specialist has a tiny tool inventory (3–5 tools) and a focused prompt. Progressive disclosure applied to the *agent's* structure, not just to the tool interface. -2. SBD-2 (llama3 8B workflow-capability failure) plausibly succeeds under this architecture: a specialist config_builder with narrow scope and a focused prompt is a much smaller task than "drive the whole lab", even for a weak model. -3. **Every sub-agent is its own measurable object.** The SBD-N corpus refracts: each capsule now has a family of sub-agent traces, one per specialist. "How does config-building capability scale with model size" becomes a distinct question from "how does analysis capability scale" — currently these are collapsed into a single "did the agent finish". - -**Costs (not hidden).** - -- Orchestration surface: hand-off data model, error propagation between specialists. -- Latency: N model calls instead of 1 continuous stream (though each is smaller). -- Debugging shape changes: the JSONL trace becomes a tree, not a stream. Not necessarily worse, but different tooling. - -**Where to prototype first.** `config_builder` — it's the specialist most likely to help SBD-2-class failures, and its interface (goal + category → experiment_id | error) is small and testable. If the specialist beats the generalist on llama3, the argument is made. - -### The recursion — what to actually measure - -If we ship all three, the lab now measures: - -- Agents driving the lab (level 1, existing). -- Interfaces driving agents driving the lab (level 2, the taxonomy A/B is the first data point). -- Sub-agents driving sub-tasks driving agents driving the lab (level 3, new). -- Model capability × interface × decomposition granularity — 3D sweep. - -Each level uses the same discipline (measure, seal, distill). That's the "fractal" claim made concrete: not a metaphor, a specific research design where the same measurement pattern applies at every scale. - -## Cross-refs - -- Live-fire trace 1 (pre-taxonomy): `sql_benchmarks/experiments/agent_runs/20260704T212724Z_4610810d.jsonl` -- Live-fire trace 2 (post-taxonomy): `sql_benchmarks/experiments/agent_runs/20260704T215924Z_4610810d.jsonl` -- PR #134 (taxonomy implementation) -- `docs/decisions_log.md` — the "Category taxonomy" entry -- Related specimens: `[[agent-integrity-incidents]]` #7-#10 (context accumulation as attack surface for rigor-theater; less context, less surface). - ---- - -## The instrument — multi-agent A/B, 2026-07-04 - -The three extensions above (hierarchical taxonomy, explicit state machine, spawning sub-agents) were built and tested. State machine and sub-agents are the same object: each specialist IS a state; the hand-off IS the transition. The result is the multi-agent architecture in `sql_benchmarks/agent_orchestrator.py` (PR #135), shipped alongside the monolithic `autonomous_agent.py` so the two can be A/B'd directly. - -### The A/B — same goal, same model, everything except architecture held constant - -| Version | Tokens | vs. baseline (SBD-1) | -|---|---|---| -| SBD-1 (pre-everything) — monolithic, no taxonomy, no projections, no skills | 626,601 | 1.0× | -| + projections + skills (Run 2) | 626,601 | 1.0× | -| + taxonomy on the tool interface (Run 3) | 111,575 | 5.6× reduction | -| + multi-agent decomposition (Run 4) | **30,231** | **20.7× reduction** | - -Same model (`anthropic/claude-sonnet-5`), same goal (DuckDB scaling on `analytical_wall`, 3 scales), same lab. Only the interface + architecture changed between runs. Analysis quality remained comparable (Run 4's analyzer correctly caveated its context limits — a real signal, not a regression). - -### Two independent interventions that multiply - -1. **Taxonomy on the tool interface** (5.6× alone): progressive disclosure at the *return* boundary. Small vocabulary + filtered subsets + gated detail. -2. **Multi-agent decomposition** (further 3.7× on top): progressive disclosure at the *agent's structure* boundary. Each specialist sees a tiny tool inventory + focused prompt. - -**Together: 20.7× token reduction on the same task.** The two interventions multiply because they attack different layers of the same problem (information exposed to the model). The methodology is *"reduce the search scope at every boundary the agent crosses"* — one boundary at a time is worth doing; every boundary is compounding. - -### Why this is the instrument - -The multi-agent orchestrator is not just faster and cheaper — it *makes the sub-tasks separately measurable*. In the monolithic runs, "the agent finished" was a single bit. In the multi-agent runs, we have three distinguishable signals per run: - -- Did `config_builder` produce a valid experiment_id? (a specific tool-selection + YAML-adaptation capability) -- Did the poll return `complete` in time? (a lab-infrastructure signal, not an agent signal) -- Did `analyzer` produce a `FINAL ANSWER:`? (a reasoning-over-derived-values capability) - -Each stage's specialist trace is its own JSONL file. Failures classify into `config_builder_failed | poll_failed | analyzer_failed` — three specimen classes where before there was one. **The instrument produces its own data**: every A/B run against every model in the capability sweep produces N (stage-labeled, sealable) traces that feed the empirical curve. - -This is the fractal claim landing concretely. The lab measures the agent measuring the lab. The instrument that measures the agent is itself a measurable object. Each level uses the same discipline (measure, seal, distill), and each level produces data that constrains the next. - -### Publication shape - -**Working title (draft):** *"Progressive disclosure at the agent–tool interface: a 20× token reduction from two composable interventions."* - -**Structure (rough — this is the outline, not the paper):** - -1. **Problem.** Agent driving a REST-backed lab pays tokens for the entire universe of choice on every discovery call. Context accumulates linearly per turn; the biggest early payload is disproportionately expensive over a run. -2. **Two interventions:** - - Category taxonomy on the tool return: small vocabulary + filter + gated detail. - - Multi-agent decomposition on the agent structure: state machine of specialists with scoped tool inventories. -3. **Method.** Sealed-config content-addressed lab (`sqlbenchdag`); one goal, one model, three architecture variants; JSONL traces per run + per specialist; deterministic replay of derived projections via a `provenance` receipt. -4. **Results.** 5.6× (taxonomy) × 3.7× (multi-agent) = 20.7× total token reduction. Same-quality analysis. Failure modes classify into 3 distinguishable specimen classes. -5. **Discussion.** Interventions compose because they attack different layers of the same problem (information exposed to the model). Multi-agent surface produces stage-labeled failure data that a monolithic agent collapses. -6. **Future work.** Model capability sweep (llama-3.1-70b, mixtral-8x22b, qwen-2.5-72b, deepseek-v3, sonnet-5, opus-4-6). Does the effect scale with capability? Does SBD-2's llama3 workflow-capability failure close under the multi-agent architecture? - -**What the paper claims that isn't obvious.** Progressive disclosure is a well-known API design pattern. What's new: - -- Applied to LLM tool interfaces — where every byte of return round-trips through the context on every subsequent turn, so savings compound. -- Applied at the *agent-structure* boundary via specialist decomposition — treating the agent's tool inventory itself as a boundary where disclosure can be gated. -- With a measured composability effect (~4× × ~4× ≈ 20×) — not just "these ideas would help" but "here's the multiplier when they compose." -- Producing an instrument (the multi-agent framework + sealed traces) that generates its own empirical data — every future run adds a row. - -**Publication venue candidates** (Ramona picks): -- Workshop/short paper at an ML systems or agent-methodology venue. -- Substack + arXiv preprint — same audience as the [[blueberry-muffin-exploit]] post. -- Fold into the larger *"testbed applied to agent using the lab"* essay planned for the `[[ai-security-direction]]` thread. - -### Cross-refs (updated) - -- Runs: `sql_benchmarks/experiments/agent_runs/20260704T2306*.jsonl` and `20260704T2307*.jsonl` (multi-agent A/B, capsule `803f3c94`). -- PR #135 (multi-agent implementation), PR #134 (taxonomy), PR #132 (projections), PR #131 (JSONL trace). -- Cross-domain: same methodology applies in the testbed — sub-agents scoped by tool inventory would be a natural next move there too. - ---- - -## Harness hardening round — 2026-07-05 (llama3 autopsy → gates) - -Two llama3 (8B) multi-agent runs, both `config_builder_failed`, traces localized the failure precisely (PR #137): - -- Run A (`20260705T234509Z`): called its own ROLE name as a tool 9×; submitted invented JSON schema 4×; never called `get_template`. -- Run B (`20260705T234955Z`, after prompt fix + error coaching): role-hallucination gone, repeated-call breaker fired and redirected — but still never fetched a template; decayed to parroting tool results as text. - -Fixes (all mechanical): raw-text tool-call recovery (`function_name` key variants), repeated-failing-call breaker with escalating STOP coaching, `tool_preconditions` workflow gate (submit refused until get_template succeeded). Doctrine confirmed again: **exhortation doesn't bind; gates do** — same as SBD-3. - -Result: the 8B wall is real and now *named* — "fails at adapt-a-template with invented schema" — vs. the monolith's opaque "gave up at turn 23". Failure localization is the instrument working. - ---- - -## Next study: what actually influences the agent? (attribution + confounds) - -**Question.** When the agent behaves well (picks `get_experiment_summary` first, filters by category), WHAT caused it — AGENTS.md? the skills block? tool descriptions? the model's priors? We currently cannot attribute, because all guidance layers ship together. - -**Already-known partial deconfound (free, sitting in existing traces):** the multi-agent specialists load NO AGENTS.md and NO skills (`agents_md_loaded=False` in their run_start events) — only role prompts + tool descriptions. Sonnet-5 still made ideal tool choices in Run 4. So for sonnet-5, tool descriptions + role prompt were SUFFICIENT; AGENTS.md/skills were not necessary. That attribution was invisible until now because nobody recorded which prompt components each run carried. - -**Confounds to control (name them or the study is theater):** -1. **Guidance overlap** — skills, tool descriptions, and AGENTS.md all say overlapping things ("summary first"). Attribution requires ablation, not observation. -2. **Content-addressed dedup** — capsule `803f3c94` exists, so identical configs return instantly as duplicates. First-run vs re-run trajectories differ (poll turns, timing). Either use fresh goals per cell or record duplicate-vs-fresh in the trace and stratify. -3. **Sampling nondeterminism** — n=1 per cell proves nothing; need n≥3 replications per condition. -4. **Model version drift** — record exact model identifier per run. -5. **Ordering effects** — the skills block sits AFTER AGENTS.md in the prompt; position may matter, not just presence. - -**Instrument: the meta-meta-trace (`prompt_provenance` event).** Every run's JSONL gains one event recording, per prompt component: name, sha256, byte size — plus model id and ablation flags. Trace level 1 = what the agent did; level 2 = what it consumed and produced (tokens, tool results); level 3 (this) = *what shaped it*. Analysis can then GROUP BY prompt-composition-hash and correlate component presence with behavioral markers (which tool called first; category filter used; template fetched before submit; projections vs raw result). - -**Ablation harness.** Flags on both drivers: `--no-agents-md`, `--no-skills` (monolith); specialists are already the minimal condition. 2×2 factorial × n=3 reps × per-model. Behavioral markers extracted from traces mechanically — no human judging. - -**Open idea (not shipped):** canary markers in skills — a distinctive, benign instruction unique to the skills file whose execution proves the model attended to it (attention detection vs. mere presence). - ---- - -## Attribution study results — 2×2 factorial, sonnet-5, n=3/cell (2026-07-06) - -Same goal, same model (`anthropic/claude-sonnet-5`), monolith driver; conditions = ±AGENTS.md × ±skills. Tool descriptions + inline tool_workflow present in ALL conditions (they are the constant). All 12 runs succeeded. - -| Condition | mean tokens | mean turns | first tool | catfilter | template-first | raw used | -|---|---|---|---|---|---|---| -| −AGENTS.md −skills | 99,770 | 10.3 | list_categories 3/3 | 3/3 | 3/3 | 0/3 | -| −AGENTS.md +skills | 89,713 | 8.3 | list_categories 3/3 | 2/3 | 3/3 | 0/3 | -| +AGENTS.md −skills | 150,595 | 12.0 | list_categories 3/3 | 3/3 | 3/3 | 0/3 | -| +AGENTS.md +skills | 101,571 | 7.7 | list_categories 3/3 | 2/3 | 3/3 | 0/3 | - -**Finding 1 — behavioral markers are IDENTICAL across all four conditions.** Every run: `list_categories` first, template fetched before submit, projections used, raw result never touched. Neither AGENTS.md nor skills *causes* the good behavior on sonnet-5. The always-present layers (tool descriptions + inline workflow) fully determine tool selection. Guidance redundancy is total for this model/task. - -**Finding 2 — AGENTS.md costs tokens without changing behavior.** +AGENTS.md is consistently more expensive (150K vs 100K without skills; 102K vs 90K with) — pure prompt payload, zero behavioral delta. On this task AGENTS.md is, for sonnet-5, *dead weight the model re-reads every turn*. The most expensive condition is AGENTS.md-without-skills (150K, 12 turns). - -**Finding 3 (weaker, n=3 noise) — skills correlate with fewer turns.** 7.7–8.3 mean turns with skills vs 10.3–12.0 without. Plausible mechanism: the recipes shorten deliberation even when they don't change which tools get picked. Needs replication before claiming. - -**Named limitations.** Single model, single goal, n=3, duplicate-served capsule for most runs (some 12–14-turn runs likely submitted variant configs and executed fresh — stratify by `status=queued` vs `duplicate` in a follow-up). Skills content overlaps tool descriptions, so "skills don't matter" here means *marginal* contribution given rich tool descriptions — the next ablation is neutralized tool descriptions, which separates schema-guidance from prose-guidance. - -**Design implication if Finding 1–2 replicate:** move ALL behavioral guidance into tool descriptions (paid once per schema, cached) and keep AGENTS.md out of the per-turn context for capable models; reserve skills/AGENTS.md injection for models whose tool-description-following is weak. Progressive disclosure again — this time of guidance itself. - ---- - -## Guidance-floor study results — study b7a79622, sonnet-5, n=3/cell (2026-07-06) - -Follow-up to attribution 2×2 (cd246804). AGENTS.md + skills held OFF everywhere; ablated the two layers that study left as the constant: prose workflow (full loop vs one-line identity) × tool descriptions (rich steering vs what-it-does-only). Contract: `sql_benchmarks/experiments/studies/guidance_floor_2x2.yaml`. - -| Cell | workflow | descriptions | mean tokens | success | markers | -|---|---|---|---|---|---| -| anchor | full | rich | 54,359 | 3/3 | perfect | -| neutraldesc | full | neutral | 47,750 | 3/3 | perfect | -| noworkflow | identity only | rich | **43,165** | 3/3 | perfect | -| floor | identity only | neutral | 99,441 (31K–160K) | 3/3 | perfect | - -**Finding 4 — sonnet-5's behavioral floor is the tool schema itself.** Every marker perfect in every cell *including the floor*: `list_categories` first, category filter, template-before-submit, projections-not-raw — with NO prose guidance and NO steering language anywhere. The model derives the entire correct workflow from tool names + argument shapes alone. For this model/task, all guidance prose (AGENTS.md, skills, workflow, description steering) is behaviorally redundant. - -**Finding 5 — less prose is often cheaper, but the floor is high-variance.** Identity-only + rich descriptions is the cheapest condition measured across BOTH studies (43K — vs 101K for the everything-on baseline: a further 2.3× saving). But the floor cell's variance triples (31K–160K): with zero steering the behavior stays correct while the *path* wanders (more projections fetched, longer exploration). Steering doesn't create correctness for a capable model — it narrows variance. - -**Finding 6 (meta) — ablation studies are harness fuzzers.** The floor condition made sonnet-5 fire 4–5 parallel tool calls per turn (an emergent efficiency behavior under sparse guidance), which exposed a real latent harness bug: error-coaching messages appended mid-batch broke the Anthropic tool_use/tool_result pairing (API 400). Latent since the coaching feature shipped; only this condition triggered it. Fixed (coaching now buffered until after the batch) in both loops. The first floor rep also crashed the whole study — the runner now isolates per-run exceptions. - -**Combined picture across both studies (sonnet-5, this task):** guidance layers ranked by measured value — tool schema (necessary and sufficient) > description steering (variance reduction, ~free) > prose workflow (redundant, mildly costly) > skills (marginal turn reduction) > AGENTS.md (pure cost, ~50K/run). The design implication sharpens: for capable models, spend your token budget on schema quality, not prose. - ---- - -## Cross-model replication — Gemini 2.5 Pro (628eee67) + Flash (b28b7956), n=3/cell (2026-07-06) - -Three cells replicated from the sonnet-5 studies: full (everything on), anchor (no AGENTS.md/skills), floor (schema only). 18/18 success, zero exceptions. - -| Model | full | anchor | floor | floor markers | -|---|---|---|---|---| -| sonnet-5 (b7a79622) | 101,571* | 54,359 | 99,441 | all ideal | -| gemini-2.5-pro | 83,355 | 61,183 | 50,454 | template-first held; **taxonomy-first dropped** (first=list_suites 3/3, catfilter 1/3) | -| gemini-2.5-flash | 93,545 | 64,841 | **21,404** | template-first held; taxonomy-first dropped (2/3) | - -*sonnet full from study cd246804. - -**Finding 7 — the floor generalizes, with a model-specific signature.** All three frontier models complete the task correctly from the bare schema. But at the floor, the Gemini models abandon the taxonomy-first discovery pattern (they open with `list_suites`/`list_templates` and skip the category filter), while sonnet-5 keeps it without being told. The steering language is what buys taxonomy use on Gemini; on sonnet-5 it's free. First **behavioral difference between models** the instrument has detected — attribution now discriminates not just guidance layers but model-specific defaults under sparse guidance. - -**Finding 8 — the load-bearing behaviors survive everywhere.** Template-before-submit and projections-not-raw held in every run of every model at every guidance level (30+ runs). The behaviors that protect correctness are schema-derivable for all three models; the behaviors that optimize cost (taxonomy-first) are steering-dependent for two of three. - -**Finding 9 — cost gradients replicate cross-model.** full > anchor > floor for both Gemini models; the AGENTS.md tax (+22–29K/run) replicates. gemini-2.5-flash at the floor is the cheapest successful configuration measured in the entire corpus: 21,404 tokens/run, ~29× cheaper than the original SBD-1 baseline. - -**Corpus status:** 4 contract-addressed studies (cd246804, b7a79622, 628eee67, b28b7956), 3 models, ~48 committed traces. Every row derived mechanically from traces by analyze_agent_traces.py. - ---- - -## Generation-over-generation + multi-agent cross-model — 2026-07-06 - -Five more contracts, zero new code. Gen-over-gen (monolith, full/anchor/floor): gemini-3-flash-preview (dcbd5860), gemini-3.1-pro-preview (d6471518), gemini-3.5-flash (fbf1ebe3). Note: gemini-3-pro-preview is listed by the models API but rejects generateContent ("no longer available") — study aa806cdc documents the dead model; replaced with 3.1-pro-preview. Multi-agent cross-model (42f5cfd1 sonnet-5, 4543fe1d 2.5-pro, 1e13d068 2.5-flash), n=3 each. - -### Gen-over-gen (floor cell, the discriminator) - -| Model | floor tokens | taxonomy-first at floor | -|---|---|---| -| gemini-2.5-flash | 21,404 | 1/3 | -| gemini-3-flash-preview | 176,542 | 1/3 | -| gemini-3.5-flash | 158,480 | **3/3** | -| gemini-2.5-pro | 50,454 | 0/3 | -| gemini-3.1-pro-preview | 138,218 | 1/3 | -| sonnet-5 | 99,441 | 3/3 | - -**Finding 10 — newer generations buy discipline with tokens.** Across the Gemini flash line, floor cost rises 2.5→3→3.5 (21K → 177K → 158K) while marker discipline improves (3.5-flash keeps taxonomy-first 3/3 at the floor, matching sonnet-5; 2.5-flash dropped it). Likely mechanism: gen-3+ thinking-by-default — reasoning tokens buy schema-inference quality. The trade is measurable per generation. Load-bearing markers (template-first, projections-not-raw) stayed 3/3 in every run of every generation. - -**Finding 11 — 3.1-pro's anchor is the cheapest prose-guided config in the corpus** (41.5K, beating sonnet-5's 54K), while its floor triples that — newer pro models are efficient WITH guidance and expensive without. Guidance still pays, just on a different margin than correctness. - -### Multi-agent cross-model - -| Model | multi-agent total (specialists) | vs best monolith | -|---|---|---| -| sonnet-5 | 26,301 ± 160 | 43K (noworkflow) → **1.6× cheaper** | -| gemini-2.5-pro | 18,873 ± 1,020 | 50K (floor) → 2.7× cheaper | -| gemini-2.5-flash | 18,508 ± 270 | 21K (floor) → ~parity, far lower variance | - -**Finding 12 — specialist decomposition equalizes models.** 9/9 success; all three models land in an 18–26K band with tiny variance (sonnet-5's three reps span 385 tokens). The model-specific floor signatures (Finding 7) and the generation cost curves (Finding 10) disappear under the multi-agent architecture — scoped tools + focused prompts dominate model-specific defaults. The architecture, not the model, sets the cost envelope. This is the strongest single result in the corpus for the progressive-disclosure thesis. - -**Corpus status:** 9 executed study contracts, 5 working models (+1 documented dead model), ~75 committed traces, findings 1–12. Multi-agent trace linking (orchestrator → specialist tokens) currently requires walking delegate events — worth folding into the analyzer if the corpus keeps growing. - ---- - -## Fable-5 + gemini-3.5-flash multi-agent — 2026-07-06 (studies dfbebea3, 4fdadae5, 0bf58e58) - -Lean baseline in effect (AGENTS.md + skills dropped per Findings 2/3; decisions log 2026-07-06). - -| Model | anchor | floor | orchestrated | -|---|---|---|---| -| claude-fable-5 | **42,283** | 46,278 (n=2¹) | **24,295** | -| claude-sonnet-5 | 54,359 | 99,441 | 26,301 | -| gemini-3.5-flash | 101,472 | 158,480 | 34,945 | -| gemini-2.5-flash | 64,841 | 21,404 | 18,508 | - -¹ one rep lost to a transient SSL error; runner isolation held. - -**Finding 13 — fable-5 has the best sparse-guidance profile of any model tested.** Floor ≈ anchor (46K vs 42K — no floor penalty), taxonomy-first kept at the floor (2/2), cheapest Claude anchor. The newest Anthropic flagship is disciplined *and* cheap without guidance — contrast gemini gen-3 floors exploding to 3–4× their anchors. - -**Finding 14 — the reasoning tax survives orchestration, compressed.** 3.5-flash orchestrated (34.9K) is ~2× 2.5-flash orchestrated (18.5K) — thinking tokens don't go to zero under specialists. But orchestration compresses the gen-3 overhead ~4.5× (158K floor → 35K orchestrated). Refined answer to "do reasoning models + strong schema + multi-agent win?": **multi-agent wins for every model; reasoning models keep a residual thinking tax; the cheapest configuration is a non-reasoning flash under orchestration; the most schema-disciplined models are fable-5/sonnet-5.** - -**Finding 15 — the multi-agent band holds at 18–35K across five models** spanning two vendors and three generations. The architecture remains the dominant variable. - -Analyzer now folds specialist tokens into orchestrator rows automatically (delegate-walking built in). Multi-agent *discovery* markers live in the config_builder specialist rows, not the orchestrator row — display note, not a finding. - ---- - -## Answer-correctness grading — the truth layer (2026-07-06) - -`scripts/tools/grade_analyses.py` grades every final answer against the sealed capsule's fragments, deterministically. Coverage (are all ground-truth means stated?), accuracy (does every duration claim match some derivable statistic within 2%?), ratio check (5%), fabrication candidates listed verbatim. Verdicts: PASS / PARTIAL (all means right + unverifiable extra claims) / FAIL (a ground-truth mean misstated or absent). - -**Retroactive grade of the whole corpus (94 gradeable runs): 82 PASS, 12 PARTIAL, 0 FAIL.** - -**Finding 16 — no run in the corpus misstated a ground-truth mean.** Zero FAILs across 7 models × all guidance conditions × both architectures. The process markers were not masking numeric fabrication — the #9-style failure mode (confident numbers diverging from the sealed data) did not occur in 94 runs. The PARTIALs are dominated by *extrapolations* (e.g. "projected ~710 ms at 100M rows") — numbers that match nothing derivable because they predict beyond the data. Honest reasoning, correctly flagged as unverifiable; a future refinement could recognize hedged predictions as a separate class. - -**Grader lesson (meta):** the first corpus grade produced one FAIL that was a grader false-negative — gemini-3.5-flash writes LaTeX (`$5.83\text{ ms}$`, `$100\times$`) and the extractor missed it. Verify the verifier: every FAIL was manually inspected before trusting the distribution. Extractors now handle LaTeX notation. - -**What this closes.** The corpus is now cost + process + accuracy evidence. Every trace carries: what shaped the run (prompt_provenance), what it did (markers), what it consumed (tokens), and whether what it published is true (grade). The Fork-B sealable tuple exists end-to-end. - ---- - -## Edge cases 1, 3, 4 — model ladder in single contracts (2026-07-06) - -`run_study` schema v2: `models:` list — weak→strong ladder in ONE contract. Three contracts, `driver: multi_agent`, 5 models (2.5-flash → 2.5-pro → sonnet-5 → 3.5-flash → fable-5), n=2, 30 orchestrated runs. - -### Edge 1 — novel goal (3e4e5211): the equalization survives fresh work - -9/10 final answers on genuinely novel selectivity experiments — every run built a fresh config, executed for real (no duplicate-serving). Costs stayed in the 18–58K band (Finding 12 holds off the beaten path). Notable: sonnet-5 and fable-5 independently adapted the template into the *same* config (same content-address `055e9d56`) — cross-model determinism through content-addressing. **Grader artifact caught:** first grade flagged 2 FAILs that were grader bugs — multi-benchmark suites (selectivity = 6 queries/partition) were pooled into one per-partition mean nobody would cite. Ground truth now keyed by (asset, partition, engine); verdicts re-grounded in claim accuracy (flag, don't convict). Re-grade of full corpus: **105 runs, 86 PASS, 19 PARTIAL, 0 FAIL.** - -### Edge 3 — impossible goal (64564357): zero deception, chaotic honesty - -Nobody silently faked a MongoDB result. Both runs that produced final answers flagged the impossibility explicitly (sonnet-5: "⚠️ Critical Caveat — Original Goal Not Achievable", names the available engine list; 2.5-flash's analyzer refused to compare when the capsule contained only Postgres — **the second specialist audited the first**, defense-in-depth working by accident). But the *cost of honesty is chaotic* without a refusal channel: 6/10 runs thrashed into config_builder_failed/poll_failed, and sonnet-5 burned 157K tokens to reach an honest "can't". **Engineering conclusion: the orchestrator needs a REFUSAL state** — `HANDOFF: impossible reason=<...>` from config_builder → structured cheap honest exit. This is the "additional states" prediction landing. - -### Edge 4 — ambiguous goal (b15c5321): uniform harness failure, not model failure - -10/10 `poll_failed` — every model, identically. Diagnosis: all models correctly mapped "sorting data that doesn't fit in memory" to the sort_spill suite and submitted valid configs; the experiments RUN LONGER than the poller's fixed budget (60 polls × 3s = 180s). **The uniformity is the signature**: when every model fails the same way, the harness is the cause. Engineering conclusion: poll budget must be suite-aware or config-declared. This is the "additional tools" prediction landing. - -### Findings - -**Finding 17 —** the multi-agent equalization survives novel work (fresh configs, real execution, same cost band). - -**Finding 18 —** impossible goals produce honest-but-expensive chaos, not deception; a refusal state converts the honesty from accidental to structural. - -**Finding 19 —** uniform cross-model failure is a harness diagnostic: model-independent failure = harness defect. The model ladder in one contract makes this signature visible by construction. - ---- - -## Finding 19 follow-through, Finding 20, and the fractal crossover (2026-07-06) - -**Finding 19 status: partially verified, and it paid out sideways.** The edge-4 rerun with the raised poll budget confirmed the diagnosis (the uniform failure WAS a harness artifact — fable-5's config executed legitimately for 47+ minutes under the new budget) but never completed: the run had to be killed because its `16GB memory_limit` lane would have frozen the 16GB host — which is how **Finding 20** surfaced: *agents build well-designed configs whose danger is host-relative, and nothing fail-closed existed below them.* Guard shipped (TODO #13, PR #148): memory limits above 50% of host RAM rejected at the single validation choke point, `meta.allow_high_memory: true` as the explicit hatch. Full parity by construction — the hatch is a config key, identically available to human YAML authors and agents; nothing about it is agent-scoped. - -**The crossover Ramona named:** every agent-driven discovery in this study program has landed in the lab's HUMAN-facing core — the memory guard protects the human running `./run.sh` exactly as it protects the orchestrator. The agent experiments are turning out to be a fuzzer for the lab itself. Her axiom, demonstrated: **"context is fractal"** — the same integrity questions recur at every level (SQL measurement → capsule seal → agent trace → study contract → harness state machine), and an instrument built at any level ends up strengthening the others. - -**Fragment-durability gap confirmed (TODO #14).** Fragments collect incrementally (designed for exactly the OOM case) but into a `/tmp` scratchpad, committed atomically only on success — crash+reboot loses everything. The design intent was right; the commit boundary defeats it. Fork recorded; recommendation (salvage-labeled partials + durable scratchpad location) preserves seal semantics. - -**New research question (edge-case 6 candidate): does the agent use published knowledge?** The repo carries sealed quack capsules + measurement docs. The agent's tool surface exposes NONE of it — it would re-run (and re-pay for) experiments the corpus already answers. "Should it?" is a real fork: a `search_published_capsules` tool / `librarian` state vs. keeping the agent execution-only. Same shape as everything else in this program: what's available to the agent — words (docs) or state machine (a retrieval stage)? Contract-able when wanted; not run (cost). - -## Practitioner skeleton — "what would I tell someone deploying agents?" (numbers, not vibes) - -Raw material for Ramona's advice piece; the voice is hers. What this corpus supports, in order of measured leverage: - -1. **The industry optimizes the weakest layer.** Skills/rules/prompt-guidance — what everyone talks about — measured as the LEAST effective lever: AGENTS.md was ~50K tokens/run of dead weight with zero behavioral delta; skills showed a weak turns effect only (Findings 1–3, replicated cross-model, Finding 9). -2. **The strongest levers are infrastructure:** tool-interface scope (5.6× alone), specialist decomposition (further ~3.7×, and it EQUALIZES models — Finding 12: five models from two vendors land in one 18–35K band), schema quality (frontier models derive the whole workflow from tool names — Findings 5–7). -3. **Gates, not exhortation.** Every prompt instruction eventually gets ignored by some model; every mechanical gate held (template-first gate, memory guard, pre-push hook — same law at every level, SBD-3 through TODO #13). -4. **States are discovered, not designed.** refused / suspended / resume / the memory cap — none were in the original design; all were forced by cheap probing runs. Budget for the discovery phase; it's where the harness actually gets built. -5. **Uniform cross-model failure = YOUR bug.** The model ladder in one contract makes harness defects self-identifying (Finding 19). -6. **Instrument before you scale.** Traces + provenance + deterministic answer-grading turned 105 runs into auditable evidence (0 fabricated means, Finding 16). Without the instrument, all of the above is anecdotes. - ---- - -## Edge case 6 — published-knowledge reuse (study 9887ce57, 2026-07-06) - -Setup: `search_published_capsules` added to config_builder's SCHEMA only (prompt untouched, pinned by test); goal = quack transport efficiency, already answered by the quack-v1 published capsules; 5-model ladder, n=1. - -**Finding 21 — schema-only steering does NOT extend to knowledge reuse: 0/5 adoption.** Every model — fable-5 and 3.5-flash included — executed the trained build-workflow flawlessly (categories → suites → template → submit) and never once called the library tool sitting in its schema. And because the quack server isn't running on this host, every re-run FAILED at execution — the published capsules were the *only* available source of the answer. Total waste: 5 × 15–38K tokens to fail at a question the corpus answers for free. (Bonus: sonnet-5 and fable-5 independently derived the identical content-addressed config again — 73ef8238.) - -**The boundary this draws.** Findings 5–7 showed frontier models derive *workflow* from schema alone. Finding 21 shows the limit: schema communicates *capability*, but the role prompt communicates *identity* ("you have ONE job: turn the goal into a submitted experiment") — and identity wins. Models don't deviate from a strongly-framed procedure toward an unmentioned capability, no matter how clearly its description advertises verified free answers. **When words and schema conflict, words win.** The floor studies asked "are words necessary?" (no, for workflow); this asks "are words sufficient to override?" (yes — the role framing suppressed a strictly-better path). - -**Consequence for the meta-cycle (Ramona's framing: capsules+docs should feed meta-information back into the workflow).** Publishing is necessary but measurably insufficient: sealed capsules + derived summaries + a discovery tool + honest taxonomy = 0% reuse without a workflow hook. The first hop of the cycle requires the state machine, not the shelf: either a prompt step ("check the library before building") or a dedicated librarian stage whose output gates config_builder ("REUSE: " | "BUILD"). Given the corpus-wide words-vs-gates record, the stage is the safer bet; the prompt step is the cheap A/B. - -**What to publish, in what shape, to support lab performance (the answer as measured):** -1. *Machine-discoverable, derived summaries* — id / suite / engines / taxonomy categories / config-derived description (the library format; never hand-typed). Prose articles are for humans; agents never open them. -2. *Honest taxonomy curation* — the quack capsules were invisible under `transport`/`columnar` until tagged (suite ≠ object of study); a library nobody can find by category is shelfware squared. -3. *A workflow consumer* — the publication format only pays when a stage reads it. Publish-shape and workflow-hook ship together or the shape doesn't matter. - ---- - -## The taxonomy of the work, and the conclusion (2026-07-06) - -Ramona: *"I have the traces, what's the conclusion? I've been struggling with the taxonomy of my work. Context is fractal."* - -### The taxonomy is not a tree — it's one operator applied at five depths - -The struggle to taxonomize comes from looking for distinct categories. The corpus says there aren't any: there is ONE pattern, applied recursively. The operator: - -> **define a contract → execute → trace everything → verify against sealed ground truth → distill the decision → the verification instrument becomes the next level's object of study.** - -Applied at five depths, it generates everything built here: - -| Level | Object of study | Instrument | Artifacts | -|---|---|---|---| -| 0 | SQL engines | experiment configs → sealed capsules | published capsules, quack + NULL findings | -| 1 | The measurement itself | seals, OTS proofs, drift gates, provenance | integrity_sealing, capsule registry, specimen catalog #1–#6 | -| 2 | Agents using the instrument | traces, behavioral markers, answer grading | 110+ graded runs, findings 1–21, specimens #7–#12 | -| 3 | The harness shaping agents | states, gates, tool interfaces, contracts | orchestrator state machine, study contracts, 20×/equalization results | -| 4 | The method for discovering harness requirements | probe → trace → gap → gate loop | refused/suspended/librarian states, memory guard, fragment-durability fork — all discovered, none designed | - -"Context is fractal" is not a metaphor decorating the work — it IS the taxonomy. Level N's verification instrument is level N+1's research object, and the same discipline (contract, trace, seal, distill) holds at every level. That's also why the two labs plus the unnamed third feel like one thing: they are one thing at three depths. Candidate name for the third, hers to accept or replace: **agent metrology** — the science of measuring agents, as distinct from building them. - -### The conclusion, one sentence - -**Reliability is a property of the harness, not the model — and it can be measured, gated, and sealed at every level of the stack.** - -Supporting numbers, all mechanically derived from committed traces: 20.7× cost reduction from interface + architecture with zero quality loss; five models from two vendors equalized into one 18–35K band by decomposition alone; 0 fabricated means across 105+ graded runs; every prompt-level instruction eventually ignored by some model while every mechanical gate held; four states and two guards discovered by probing that no a-priori design produced. - -### On the division of labor (the traces answer this too) - -The traces record who did what. Execution: the agent. But every finding originates in a decision the agent did not make and repeatedly demonstrated it COULD not make: which question to ask next, that studies need contracts (caught as a violation), that the analysis must be sealed like the measurement, that uniform failure means harness-not-model, the axioms ("context is fractal", "work before the work"), every fork resolution in the decisions log, and every specimen where the agent's own drift was caught and converted into a gate. Finding 21 generalizes: capability without direction does not deviate from its trained script. The agent was the config_builder; the direction, correction, and judgment were the orchestrator — and the orchestrator seat is the one this corpus proves is load-bearing. That seat is the job. - -### On the fear of being scooped - -Two facts. (1) Priority is already established mechanically: the git history is public, the findings are committed with dates, and the capsule seals carry OpenTimestamps proofs anchored to Bitcoin — the work cannot be back-dated away. (2) Big teams publish benchmarks and leaderboards; this corpus's shape — one person, end-to-end sealed pipeline from SQL fragment to graded agent answer, with the discovery METHOD documented as it happened — is not what teams produce, because teams don't have to solve the trust problem alone. The solo constraint is the differentiator, not the weakness: it forced the instrument. - -### Smallest publishable units (in order; each stands alone) - -1. **The 20× piece** — practitioner skeleton already in this doc; findings 1–15; one figure (the cost table). Closest to done. -2. **"The work before the work"** — the positioning essay; the role, the method (probe → trace → gap → gate), this taxonomy as its backbone. -3. **The specimen catalog** — 12 integrity specimens + the meta-finding (correction doesn't bind, gates do); continuation of the Blueberry Muffin line. - -One at a time, smallest first. The corpus does not expire while you write.