rctruta · rctruta · Jul 6, 2026 · Jul 6, 2026 · Jul 6, 2026
diff --git a/TODO.md b/TODO.md
@@ -190,3 +190,9 @@ Session-start and session-end records with model, goal, outcome. Written to `--l
 ## 11. agent_runs/ is a flat pile — restructure as <run_id>/ directories
 
 **STATUS: OPEN** — surfaced 2026-07-06 (Ramona). All traces live flat in `sql_benchmarks/experiments/agent_runs/*.jsonl`. Target shape: `agent_runs/<id>/` holding everything pertaining to that run (trace, and later: sealed analysis, per-run manifest), mirroring the capsule convention in `results/<id>/`. For multi-agent runs the orchestrator's id is the natural directory; specialist traces live inside it. Deferred by choice — corpus collection takes priority; do the reorg before the corpus gets big enough for the move to be painful (analyzer + committed-trace paths need a migration in the same PR).
+
+## 12. Stale running markers — a crashed executor blocked resubmission forever
+
+**STATUS: SHIPPED 2026-07-06** — `has_running_marker` is now liveness-aware. Observed live: a session teardown killed the API mid-execution of `209fc5df`, leaving an orphaned `running.json`; `check_registry` would have answered "already running — poll instead" to every resubmission of that config, permanently, and the marker had to be removed by hand.
+
+Fix (single choke point — `running_marker.has_running_marker`, inherited by `check_registry` and the API's `is_running`): a marker only counts as running if it is readable, younger than `MAX_MARKER_AGE_SECONDS` (6h — also caps the PID-reuse window), and (same-host) its recorded PID is alive. Stale markers are removed on detection (self-heal) with a loud `[WARN]`. Different-host markers within the age window are conservatively treated as alive (can't probe a remote PID). 6 regression tests pin each rule, including the exact 209fc5df scenario (dead PID → not running → marker gone).
diff --git a/docs/decisions_log.md b/docs/decisions_log.md
@@ -336,3 +336,17 @@ Human-and-agent parity: `sqlbench project means <id>` (CLI),
 **Decision (own repo?).** The harness (`agent_tools/specialist/orchestrator/trace`, `run_study`, analyzer, grader) stays IN this repo until a second consumer exists. Extraction trigger: ai-security-testbed importing it, or the article requiring a citable standalone artifact. Extracting now would freeze the API exactly while the edge-case studies are telling us which states/tools are missing (e.g., the refusal state that edge-3 is expected to surface).
 
 **Cross-refs.** Edge-case contracts `edge1_novel_goal`, `edge3_adversarial_goal`, `edge4_ambiguous_goal`; run_study schema v2 (`models:` list — weak→strong ladder in one contract).
+
+---
+
+## 2026-07-06 — SUSPENDED state + resume: the capsule is the durable hand-off
+
+**Decision.** Poll timeout while the experiment is still executing is now `suspended`, not `poll_failed` — a first-class non-error outcome carrying the experiment_id. `Orchestrator.resume(exp_id)` (CLI: `multi_agent.py --resume <id>`) picks up at the analyzer once the capsule completes, for a few thousand tokens. Also this session: structured REFUSAL state (`HANDOFF: impossible reason=…`), contract-declared `poll_budget_seconds`, and liveness-aware running markers (TODO #12 — self-healing on dead PID / over-age / corrupt).
+
+**Fork closed.** Tuning poll budgets per suite. Edge-4's rerun proved no budget is "right": fable-5 correctly chose a 16GB out-of-memory sort that outlived even a 30-minute budget *while executing legitimately*. Synchronous waiting conflates "slow" with "dead".
+
+**Fork opened.** Long-running experiments become first-class: submit → suspend → resume across sessions or crashes. The capsule (content-addressed, server-side, durable) is the hand-off point — the same object that already anchors reproducibility now anchors continuity. Pairs with the liveness-aware markers: "dead" is now detected by evidence (PID, age), never inferred from silence.
+
+**Live validation of the recovery story (same day).** A session teardown killed the runner mid-poll and the API mid-execution. Traces survived (append-per-event JSONL); the stale running marker was the only manual surgery — and that class is now self-healing.
+
+**Cross-refs.** TODO #12, edge-3/edge-4 studies, Finding 18/19, `[[work-before-the-work]]` (states uncovered by pre-deployment probing — exactly the deliverable of the role Ramona is naming).
diff --git a/scripts/multi_agent.py b/scripts/multi_agent.py
@@ -42,16 +42,24 @@ def main():
                         help="litellm model identifier")
     parser.add_argument("--goal", type=str, required=True,
                         help="Natural-language goal for the orchestrator")
+    parser.add_argument("--resume", metavar="EXP_ID",
+                        help="Resume a SUSPENDED run: skip config_builder, poll "
+                             "this experiment_id, then run the analyzer. The "
+                             "goal is still required (the analyzer needs it).")
+    parser.add_argument("--poll-budget-seconds", type=float, default=180.0,
+                        help="How long the poll stage waits before suspending.")
     args = parser.parse_args()
 
     console.print(Panel(f"[bold cyan]GOAL:[/bold cyan] {args.goal}",
                         title="🤖 Multi-Agent Orchestrator"))
-    console.print(f"[dim]Model: {args.model}[/dim]")
+    console.print(f"[dim]Model: {args.model}"
+                  + (f" | resuming {args.resume}" if args.resume else "") + "[/dim]")
 
-    orch = Orchestrator(goal=args.goal, model=args.model)
+    orch = Orchestrator(goal=args.goal, model=args.model,
+                        poll_budget_seconds=args.poll_budget_seconds)
     console.print(f"[dim]Orchestrator trace: {orch.trace.path}[/dim]\n")
 
-    result = orch.run()
+    result = orch.resume(args.resume) if args.resume else orch.run()
 
     console.print(f"\n[bold]Outcome:[/bold] {result.outcome}")
     console.print(f"[dim]experiment_id: {result.experiment_id}[/dim]")
@@ -61,6 +69,13 @@ def main():
     if result.outcome == "complete" and result.analysis:
         console.print(Panel(Markdown(result.analysis), title="📊 Analysis",
                             border_style="green"))
+    elif result.outcome == "suspended":
+        console.print(
+            f"\n[yellow]SUSPENDED[/yellow] — experiment {result.experiment_id} is still "
+            f"executing server-side. Resume any time with:\n"
+            f"  python scripts/multi_agent.py --resume {result.experiment_id} "
+            f"--goal <original goal> --model {args.model}"
+        )
     elif result.error:
         console.print(f"\n[bold red]Error:[/bold red] {result.error}")
         sys.exit(1)

diff --git a/scripts/run_study.py b/scripts/run_study.py
@@ -83,7 +83,9 @@ def run_cell_rep(contract: dict, study_id: str, cell_name: str, rep: int,
         return "ran"  # run_agent prints its own outcome; trace has run_end
     else:  # multi_agent
         from sql_benchmarks.agent_orchestrator import Orchestrator
-        orch = Orchestrator(goal=contract["goal"], model=model)
+        orch = Orchestrator(goal=contract["goal"], model=model,
+                            poll_budget_seconds=float(
+                                contract.get("poll_budget_seconds", 180)))
         orch.trace.prompt_provenance(components={}, ablation_flags={
             "architecture": "orchestrator", **study_stamp})
         result = orch.run()

diff --git a/sql_benchmarks/agent_orchestrator.py b/sql_benchmarks/agent_orchestrator.py
@@ -52,6 +52,16 @@
 
     HANDOFF: experiment_id=<8-hex>
 
+If the goal CANNOT be satisfied with the engines and suites this lab
+actually has (e.g. it names a database that is not in the catalog, or
+asks for a workload no suite covers), DO NOT substitute something else
+and do not force a submission. Produce instead:
+
+    HANDOFF: impossible reason=<one line naming exactly what is missing>
+
+That is a successful outcome — an honest refusal is worth more than a
+silently substituted experiment.
+
 That signals you're done. Do not analyze results — that's a different
 specialist. Do not produce a narrative — just the HANDOFF line.
 """
@@ -121,13 +131,20 @@
 
 
 _HANDOFF_RE = re.compile(r"HANDOFF:\s*experiment_id=([0-9a-f]{8})", re.IGNORECASE)
+_REFUSAL_RE = re.compile(r"HANDOFF:\s*impossible\s+reason=(.+)", re.IGNORECASE)
 
 
 def _parse_handoff(content: str) -> Optional[dict]:
-    """The config_builder signals completion with `HANDOFF: experiment_id=<8hex>`."""
+    """config_builder completion signals: `HANDOFF: experiment_id=<8hex>`
+    (built) or `HANDOFF: impossible reason=<text>` (structured refusal —
+    the state edge-3 showed was missing: without it, honest models thrash
+    through mechanical failures to say 'can't')."""
     m = _HANDOFF_RE.search(content or "")
     if m:
         return {"experiment_id": m.group(1)}
+    r = _REFUSAL_RE.search(content or "")
+    if r:
+        return {"impossible": r.group(1).strip()}
     return None
 
 
@@ -167,7 +184,7 @@ def poll_until_terminal(experiment_id: str,
 
 @dataclass
 class OrchestratorResult:
-    outcome: str  # "complete" | "config_builder_failed" | "poll_failed" | "analyzer_failed"
+    outcome: str  # "complete" | "refused" | "suspended" | "config_builder_failed" | "poll_failed" | "analyzer_failed"
     experiment_id: Optional[str]
     analysis: Optional[str]
     orchestrator_run_id: str
@@ -176,16 +193,24 @@ class OrchestratorResult:
 
 
 class Orchestrator:
-    """Runs the state machine: config_builder → poll → analyzer.
+    """Runs the state machine: config_builder → poll → analyzer, with a
+    REFUSED terminal state (config_builder judged the goal unsatisfiable
+    with the lab's actual engines/suites — a first-class honest outcome,
+    not a failure).
 
     Each stage's specialist gets its own JSONL trace. The orchestrator's
     own trace records `delegate` events pointing at each specialist's
     `sub_run_id`, so a reader can walk the multi-agent tree by opening
-    the orchestrator trace and following the sub_run_ids."""
+    the orchestrator trace and following the sub_run_ids.
+
+    `poll_budget_seconds` is contract-declarable: edge-4 showed the fixed
+    180s default silently kills legitimately slow suites (sort_spill) —
+    uniform cross-model 'failure' that was actually a harness defect."""
 
-    def __init__(self, goal: str, model: str):
+    def __init__(self, goal: str, model: str, poll_budget_seconds: float = 180.0):
         self.goal = goal
         self.model = model
+        self.poll_budget_seconds = poll_budget_seconds
         self.trace = AgentTrace(
             goal=f"[orchestrator] {goal}", model=model,
             agents_md_loaded=False, max_turns=0,
@@ -202,6 +227,15 @@ def run(self) -> OrchestratorResult:
             input_summary=self.goal[:200], outcome=cb.outcome,
             output_summary=(json.dumps(cb.output) if cb.output else None),
         )
+        if cb.outcome == "ok" and cb.output and "impossible" in cb.output:
+            # Structured refusal — terminal, honest, cheap. Not a failure.
+            reason = cb.output["impossible"]
+            self.trace.run_end("refused", turns_used=0, error=None)
+            return OrchestratorResult(
+                outcome="refused", experiment_id=None,
+                analysis=f"REFUSED: {reason}",
+                orchestrator_run_id=self.trace.run_id, sub_run_ids=sub_ids,
+            )
         if cb.outcome != "ok" or not cb.output or "experiment_id" not in cb.output:
             self.trace.run_end("config_builder_failed", turns_used=0, error=cb.error)
             return OrchestratorResult(
@@ -211,23 +245,79 @@ def run(self) -> OrchestratorResult:
             )
         exp_id = cb.output["experiment_id"]
 
-        # Stage 2: pure-Python poller
-        poll_result = poll_until_terminal(exp_id)
+        # Stage 2: pure-Python poller (budget contract-declarable)
+        interval = 3.0
+        poll_result = poll_until_terminal(
+            exp_id,
+            max_polls=max(1, int(self.poll_budget_seconds / interval)),
+            interval_seconds=interval,
+        )
         self.trace.delegate(
             stage="poll", sub_run_id=None,
             input_summary=f"experiment_id={exp_id}",
             outcome=poll_result["status"],
             output_summary=json.dumps(poll_result),
         )
+        if poll_result["status"] == "timeout":
+            # SUSPENDED, not failed: the experiment is still executing
+            # server-side and will finish regardless of who's watching.
+            # No poll budget is "right" for open-ended workloads (edge-4:
+            # a legitimate 16GB out-of-memory sort outlived a 30-minute
+            # budget). The run is resumable: `Orchestrator.resume(exp_id)`
+            # picks up at the analyzer for a few thousand tokens once the
+            # capsule completes.
+            self.trace.run_end("suspended", turns_used=0, error=None)
+            return OrchestratorResult(
+                outcome="suspended", experiment_id=exp_id, analysis=None,
+                orchestrator_run_id=self.trace.run_id, sub_run_ids=sub_ids,
+                error=None,
+            )
+        if poll_result["status"] != "complete":
+            self.trace.run_end("poll_failed", turns_used=0, error=str(poll_result))
+            return OrchestratorResult(
+                outcome="poll_failed", experiment_id=exp_id, analysis=None,
+                orchestrator_run_id=self.trace.run_id, sub_run_ids=sub_ids,
+                error=f"poll returned {poll_result['status']}: {poll_result.get('detail') or poll_result.get('error')}",
+            )
+
+        return self._analyzer_stage(exp_id, sub_ids)
+
+    def resume(self, exp_id: str) -> OrchestratorResult:
+        """Pick up a SUSPENDED run: skip config_builder (the experiment
+        already exists), check/poll the capsule, then run the analyzer.
+        Cheap — the analyzer is a few thousand tokens; polling is free.
+
+        Callable any time after suspension: minutes later, next session,
+        or after a crash — the capsule is the durable hand-off point."""
+        sub_ids: dict = {"config_builder": None}  # skipped on resume
+        interval = 3.0
+        poll_result = poll_until_terminal(
+            exp_id,
+            max_polls=max(1, int(self.poll_budget_seconds / interval)),
+            interval_seconds=interval,
+        )
+        self.trace.delegate(
+            stage="poll", sub_run_id=None,
+            input_summary=f"experiment_id={exp_id} (resume)",
+            outcome=poll_result["status"],
+            output_summary=json.dumps(poll_result),
+        )
+        if poll_result["status"] == "timeout":
+            self.trace.run_end("suspended", turns_used=0, error=None)
+            return OrchestratorResult(
+                outcome="suspended", experiment_id=exp_id, analysis=None,
+                orchestrator_run_id=self.trace.run_id, sub_run_ids=sub_ids,
+            )
         if poll_result["status"] != "complete":
             self.trace.run_end("poll_failed", turns_used=0, error=str(poll_result))
             return OrchestratorResult(
                 outcome="poll_failed", experiment_id=exp_id, analysis=None,
                 orchestrator_run_id=self.trace.run_id, sub_run_ids=sub_ids,
                 error=f"poll returned {poll_result['status']}: {poll_result.get('detail') or poll_result.get('error')}",
             )
+        return self._analyzer_stage(exp_id, sub_ids)
 
-        # Stage 3: analyzer
+    def _analyzer_stage(self, exp_id: str, sub_ids: dict) -> OrchestratorResult:
         # Fetch the config's `definitions` block server-side (no LLM tokens)
         # so the analyzer knows what the partition labels MEAN (row counts).
         # Run-4 lesson: without this the analyzer had to assume "large is

diff --git a/sql_benchmarks/experiments/studies/edge4_rerun_budget.yaml b/sql_benchmarks/experiments/studies/edge4_rerun_budget.yaml
@@ -0,0 +1,28 @@
+# Edge 4 RERUN with the two states shipped after the first execution:
+#   - poll_budget_seconds (contract-declared; first run died at the fixed
+#     180s default while sort_spill was still executing — Finding 19)
+#   - refusal state exists but should NOT trigger here (goal is valid).
+#
+# Cost-conscious design (Ramona, 2026-07-06): n=1 per model, two models
+# only — this verifies a GATE, not a distribution. Replication buys
+# nothing for defect-fix verification.
+
+meta:
+  name: "Edge 4 rerun — suite-aware poll budget verification"
+  description: >
+    Same ambiguous sort_spill goal; poll budget raised to 30 min.
+    Success = poll survives to terminal state and analyzer produces a
+    graded answer.
+
+driver: multi_agent
+models: [anthropic/claude-fable-5, gemini/gemini-2.5-flash]
+replications: 1
+poll_budget_seconds: 1800
+
+goal: >
+  I have a feeling DuckDB gets much slower when it has to sort data that
+  doesn't fit in memory. Can you check whether that's true?
+
+cells:
+  orchestrated:
+    flags: {}
diff --git a/sql_benchmarks/running_marker.py b/sql_benchmarks/running_marker.py
@@ -69,8 +69,67 @@ def read_running_marker(results_dir: str, exp_id: str) -> Optional[dict]:
         return None
 
 
-def has_running_marker(results_dir: str, exp_id: str) -> bool:
-    return os.path.exists(marker_path(results_dir, exp_id))
+# A run older than this is presumed dead regardless of PID state — also
+# bounds the PID-reuse false-alive window. Generous: real experiments run
+# minutes to ~an hour.
+MAX_MARKER_AGE_SECONDS = 6 * 3600
+
+
+def _pid_alive(pid) -> bool:
+    try:
+        os.kill(pid, 0)
+        return True
+    except ProcessLookupError:
+        return False
+    except PermissionError:
+        return True  # exists, owned by someone else
+    except (OverflowError, ValueError, TypeError):
+        return False  # garbage pid in the marker
+
+
+def has_running_marker(results_dir: str, exp_id: str,
+                       max_age_seconds: float = MAX_MARKER_AGE_SECONDS) -> bool:
+    """True only if the marker exists AND the run it describes is plausibly
+    alive. A crashed executor (killed API process, torn-down session) leaves
+    an orphaned marker that would otherwise block resubmission of the same
+    config forever — observed live 2026-07-06 (capsule 209fc5df: session
+    teardown killed the API mid-execution; the stale marker had to be
+    removed by hand). See TODO.md #12.
+
+    Staleness rules, in order:
+      - unreadable/corrupt marker            -> stale
+      - older than max_age_seconds           -> stale (any host; also caps
+                                                the PID-reuse window)
+      - same host and recorded PID not alive -> stale
+      - different host, within age           -> assumed alive (can't probe)
+
+    Stale markers are REMOVED (self-heal) with a loud warning, so the
+    status endpoint and check_registry recover without operator surgery."""
+    payload = read_running_marker(results_dir, exp_id)
+    if payload is None:
+        # Missing entirely, or unreadable. If the file exists but can't be
+        # parsed, it can't testify that anything is running — remove it.
+        if os.path.exists(marker_path(results_dir, exp_id)):
+            print(f"[WARN] corrupt running marker for {exp_id} — removing (stale)")
+            clear_running_marker(results_dir, exp_id)
+        return False
+
+    age = time.time() - float(payload.get("started_at") or 0)
+    if age > max_age_seconds:
+        print(f"[WARN] running marker for {exp_id} is {age/3600:.1f}h old "
+              f"(max {max_age_seconds/3600:.1f}h) — presumed dead, removing")
+        clear_running_marker(results_dir, exp_id)
+        return False
+
+    if payload.get("hostname") == socket.gethostname():
+        pid = payload.get("pid")
+        if not _pid_alive(pid):
+            print(f"[WARN] running marker for {exp_id} names dead pid {pid} — "
+                  "executor crashed; removing stale marker")
+            clear_running_marker(results_dir, exp_id)
+            return False
+
+    return True
 
 
 def clear_running_marker(results_dir: str, exp_id: str) -> None: