feat: refusal + suspended/resume states, contract poll budget, self-healing markers#147
Merged
Conversation
…cks resubmission (TODO #12) Observed live 2026-07-06: session teardown killed the API mid-execution of 209fc5df, leaving an orphaned running.json. check_registry would have refused every resubmission of that config forever ("already running — poll instead"); the marker had to be removed by hand. has_running_marker is now liveness-aware (single choke point; check_registry and the API's is_running inherit): - unreadable/corrupt marker -> stale - older than MAX_MARKER_AGE_SECONDS (6h; also caps PID-reuse window) -> stale - same host + recorded PID dead -> stale - different host within age -> conservatively alive (can't probe) Stale markers are removed on detection (self-heal) with a loud [WARN]. 6 regression tests incl. the exact 209fc5df scenario.
Edge-4's rerun proved no poll budget is "right" for open-ended workloads: fable-5 correctly chose a 16GB out-of-memory sort that outlived even a 30-minute budget while executing legitimately. Synchronous waiting conflates "slow" with "dead". - Poll timeout while the experiment still runs -> outcome "suspended" (non-error, experiment_id preserved). "poll_failed" now means the experiment actually failed or polling errored. - Orchestrator.resume(exp_id): skip config_builder, poll, run analyzer. Callable minutes later, next session, or after a crash — the content-addressed capsule is the durable hand-off point. - CLI: multi_agent.py --resume <exp_id> + --poll-budget-seconds. 4 new tests (31 total): suspend-not-fail on timeout, resume completes when ready, resume re-suspends while running, resume surfaces real execution failures. Ships with (same branch): structured REFUSAL state, contract-declared poll_budget_seconds, liveness-aware self-healing running markers (TODO #12). Decisions-log entry added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two states edge cases 3+4 demanded, plus the crash-recovery fix from today's live incident.
HANDOFF: impossible reason=…from config_builder → orchestrator outcomerefused(first-class honest exit, not a failure). Edge-3 evidence: honest models burned up to 157K tokens thrashing to say "can't".suspended(non-error, experiment_id preserved);Orchestrator.resume(exp_id)/multi_agent.py --resume <id>picks up at the analyzer later. Edge-4 rerun evidence: a legitimate 16GB out-of-memory sort outlived even a 30-min budget — no synchronous budget is "right"; the content-addressed capsule is the durable hand-off.poll_budget_seconds(edge-4's original defect: fixed 180s).209fc5df, orphaned marker would have blocked resubmission forever.has_running_markernow self-heals on dead PID / over-age / corrupt, with loud warnings.Tests