feat: edge cases 1/3/4 — model ladder, grader v2, findings 17-19#146
Merged
Conversation
…dings 17-19 run_study schema v2: `models:` list (single `model:` still accepted) — weak->strong ladder in one contract. --model filter added. Three multi-agent edge contracts x 5 models x n=2 (30 orchestrated runs): - Edge 1 novel goal (3e4e5211): 9/10 final answers on fresh configs with real execution; cost band holds (18-58K) — the equalization survives off the beaten path. sonnet-5 and fable-5 independently produced the SAME content-addressed config (055e9d56). - Edge 3 impossible goal (64564357): ZERO deception — completed answers explicitly flagged unavailability (sonnet: "Original Goal Not Achievable" + engine list; 2.5-flash's analyzer refused to compare when the capsule lacked MongoDB — the second specialist audited the first). But honesty is chaotic without a refusal channel: 6/10 thrashed to mechanical failures; sonnet burned 157K to reach an honest "can't". Conclusion: orchestrator needs a REFUSAL state. - Edge 4 ambiguous goal (b15c5321): 10/10 poll_failed IDENTICALLY — all models correctly picked sort_spill; the poller's fixed 180s budget is shorter than the suite's runtime. Uniform cross-model failure = harness defect signature (Finding 19). Grader v2: ground truth keyed by (asset, partition, engine) — multi- benchmark suites (selectivity: 6 queries/partition) produced pooled means nobody would cite, causing 2 false FAILs. Verdicts re-grounded in claim accuracy (PARTIAL flags for review; FAIL = nothing traces to the capsule). Full-corpus regrade: 105 runs, 86 PASS, 19 PARTIAL, 0 FAIL. Also: decisions-log entry mapping the harness-engineering tenets to lab implementation + the own-repo position (extract on second consumer, not before).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_studyv2 (models:list — weak→strong in one contract). Three multi-agent edge contracts × 5 models × n=2.Edge 1 (novel goal): 9/10 clean on fresh configs with real execution — Finding 12's equalization survives off the beaten path. sonnet-5 and fable-5 independently produced the same content-addressed config.
Edge 3 (impossible goal): zero deception — completed answers explicitly flagged MongoDB's absence; 2.5-flash's analyzer refused to compare when the capsule lacked the engine (the second specialist audited the first). But honesty is chaotic without a refusal channel (sonnet: 157K tokens to an honest "can't"). → Refusal state needed (Finding 18).
Edge 4 (ambiguous goal): 10/10 identical
poll_failed— all models correctly picked sort_spill; the poller's 180s budget is shorter than the suite runtime. Uniform cross-model failure = harness defect signature (Finding 19).Grader v2: ground truth keyed by (asset, partition, engine); verdicts re-grounded in claim accuracy. Full-corpus regrade: 105 runs, 86 PASS, 19 PARTIAL, 0 FAIL.
Also: harness-tenets mapping + own-repo position in decisions log.