From 3f80d731be8c7237f5a9b94a26b57a5cd2ac8144 Mon Sep 17 00:00:00 2001 From: rctruta <17259677+rctruta@users.noreply.github.com> Date: Mon, 6 Jul 2026 10:18:49 -0400 Subject: [PATCH] docs: TODO #14 (fragment durability fork), Finding 19/20 closure, fractal crossover, practitioner skeleton MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - TODO #14: confirmed the atomic scratchpad-commit nullifies incremental fragment collection on crash (fragments live in /tmp mkdtemp, committed only at successful finalization; 47 min of 209fc5df -> zero durable fragments). Design fork recorded — Ramona's call; recommendation (b)+(c) preserves seal semantics while adding crash durability. - Finding 19 recorded as partially verified: budget fix confirmed the uniform failure was a harness artifact, but the rerun surfaced Finding 20 (host-relative danger in well-designed configs -> TODO #13 guard, full human/agent parity via the shared validation choke point). - Ramona's axiom recorded: "context is fractal" — agent-driven discoveries keep landing in the human-facing lab core. - Edge-case 6 candidate: does the agent use published capsules/docs? (librarian state vs execution-only — words or state machine, again). - Practitioner-advice skeleton: the corpus-backed inversion — everyone pushes skills/rules; the measured leverage is harness infrastructure. --- TODO.md | 11 +++++++++++ scratch/reducing_agent_search_scope.md | 23 +++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/TODO.md b/TODO.md index 758a163..8e74650 100644 --- a/TODO.md +++ b/TODO.md @@ -204,3 +204,14 @@ Fix (single choke point — `running_marker.has_running_marker`, inherited by `c Guard shipped at the single validation choke point (`_check_memory_limits_fit_host`): any engine memory-limit (matrix lane or engine_params) above 50% of physical RAM is rejected at submission, with `meta.allow_high_memory: true` as the explicit, loud override. 6 tests incl. the exact freeze config. The agent coaching loop turns the 422 into a smaller re-submission automatically. Still open (harness-tenets gap, deliberate): OS-level enforcement (sandbox/cgroup/ulimit) for defense-in-depth below the validation layer — validation can't stop a workload that grows past its declared limit. + +## 14. Partial fragments die with the scratchpad — atomic commit nullifies incremental collection on crash + +**STATUS: OPEN — design fork is Ramona's.** Confirmed 2026-07-06 while investigating the OOM freeze: fragments ARE written incrementally per benchmark (the original design intent — survive partial failure), but into an isolated scratchpad (`harness.py: tempfile.mkdtemp` under system `/tmp`), and only committed to canonical `results//` atomically at successful finalization (`coordinator.py` verify-then-commit). Consequence: any crash/OOM/kill mid-run orphans the scratchpad — and a freeze+reboot erases `/tmp` entirely. 47 minutes of the killed `209fc5df` run produced zero durable fragments. The incremental-collection design is intact; its crash-durability benefit is nullified by WHERE it collects and WHEN it commits. + +Options (fork, not decided): +- (a) Commit fragments incrementally to canonical — breaks scratchpad isolation and the verify-then-commit integrity discipline (drift gate exists for a reason). +- (b) Keep atomic commit; add a salvage path — on failure detection, copy scratchpad fragments to `results//partial_fragments/`, loudly labeled non-canonical. Partial data becomes recoverable but can never masquerade as a sealed capsule. +- (c) Move the scratchpad off `/tmp` to a repo-local gitignored dir — survives reboot; orphans become findable/salvageable after any crash. + +Recommendation: (b) + (c) together — preserves every integrity property, adds crash durability. (a) is the only one that compromises the seal semantics. diff --git a/scratch/reducing_agent_search_scope.md b/scratch/reducing_agent_search_scope.md index 312219d..1ff61a8 100644 --- a/scratch/reducing_agent_search_scope.md +++ b/scratch/reducing_agent_search_scope.md @@ -403,3 +403,26 @@ Nobody silently faked a MongoDB result. Both runs that produced final answers fl **Finding 18 —** impossible goals produce honest-but-expensive chaos, not deception; a refusal state converts the honesty from accidental to structural. **Finding 19 —** uniform cross-model failure is a harness diagnostic: model-independent failure = harness defect. The model ladder in one contract makes this signature visible by construction. + +--- + +## Finding 19 follow-through, Finding 20, and the fractal crossover (2026-07-06) + +**Finding 19 status: partially verified, and it paid out sideways.** The edge-4 rerun with the raised poll budget confirmed the diagnosis (the uniform failure WAS a harness artifact — fable-5's config executed legitimately for 47+ minutes under the new budget) but never completed: the run had to be killed because its `16GB memory_limit` lane would have frozen the 16GB host — which is how **Finding 20** surfaced: *agents build well-designed configs whose danger is host-relative, and nothing fail-closed existed below them.* Guard shipped (TODO #13, PR #148): memory limits above 50% of host RAM rejected at the single validation choke point, `meta.allow_high_memory: true` as the explicit hatch. Full parity by construction — the hatch is a config key, identically available to human YAML authors and agents; nothing about it is agent-scoped. + +**The crossover Ramona named:** every agent-driven discovery in this study program has landed in the lab's HUMAN-facing core — the memory guard protects the human running `./run.sh` exactly as it protects the orchestrator. The agent experiments are turning out to be a fuzzer for the lab itself. Her axiom, demonstrated: **"context is fractal"** — the same integrity questions recur at every level (SQL measurement → capsule seal → agent trace → study contract → harness state machine), and an instrument built at any level ends up strengthening the others. + +**Fragment-durability gap confirmed (TODO #14).** Fragments collect incrementally (designed for exactly the OOM case) but into a `/tmp` scratchpad, committed atomically only on success — crash+reboot loses everything. The design intent was right; the commit boundary defeats it. Fork recorded; recommendation (salvage-labeled partials + durable scratchpad location) preserves seal semantics. + +**New research question (edge-case 6 candidate): does the agent use published knowledge?** The repo carries sealed quack capsules + measurement docs. The agent's tool surface exposes NONE of it — it would re-run (and re-pay for) experiments the corpus already answers. "Should it?" is a real fork: a `search_published_capsules` tool / `librarian` state vs. keeping the agent execution-only. Same shape as everything else in this program: what's available to the agent — words (docs) or state machine (a retrieval stage)? Contract-able when wanted; not run (cost). + +## Practitioner skeleton — "what would I tell someone deploying agents?" (numbers, not vibes) + +Raw material for Ramona's advice piece; the voice is hers. What this corpus supports, in order of measured leverage: + +1. **The industry optimizes the weakest layer.** Skills/rules/prompt-guidance — what everyone talks about — measured as the LEAST effective lever: AGENTS.md was ~50K tokens/run of dead weight with zero behavioral delta; skills showed a weak turns effect only (Findings 1–3, replicated cross-model, Finding 9). +2. **The strongest levers are infrastructure:** tool-interface scope (5.6× alone), specialist decomposition (further ~3.7×, and it EQUALIZES models — Finding 12: five models from two vendors land in one 18–35K band), schema quality (frontier models derive the whole workflow from tool names — Findings 5–7). +3. **Gates, not exhortation.** Every prompt instruction eventually gets ignored by some model; every mechanical gate held (template-first gate, memory guard, pre-push hook — same law at every level, SBD-3 through TODO #13). +4. **States are discovered, not designed.** refused / suspended / resume / the memory cap — none were in the original design; all were forced by cheap probing runs. Budget for the discovery phase; it's where the harness actually gets built. +5. **Uniform cross-model failure = YOUR bug.** The model ladder in one contract makes harness defects self-identifying (Finding 19). +6. **Instrument before you scale.** Traces + provenance + deterministic answer-grading turned 105 runs into auditable evidence (0 fabricated means, Finding 16). Without the instrument, all of the above is anecdotes.