[babaliauskas]

Why

max_latency_increase flipped the verdict of one real suite between two runs. The suite was 21 examples (Sonnet 4.6 → 5.5) with a 0.30 budget:

	run 9b78e4	run 13d053
old: mean target ÷ mean source − 1	+27.2% ✅	+35.6% ❌ (only failing row)
median of per-example ratios	+11.4%	+23.7%
new: 20%-trimmed geometric mean of per-example ratios	+13.2% ✅	+21.6% ✅

In both runs, 86–95% of the excess came from 2–3 target calls of 3–5 s. They landed on different examples each run, and the same examples took 1.1–1.9 s in the other run. That is provider tail noise.

What changes

• The unit is the example pair. Calls are grouped by (prompt_id, example_id), and an example counts only when each side has an error-free call. An example the target failed on drops out of both sides, so the source's slowest example is no longer compared against a target that never answered it.
• Latency is gated on the 20%-trimmed geometric mean of target/source ratios. The trim drops floor(0.2·n) pairs from each end of the sorted log-ratios.
  • Why not the median: in a simulation that adds a real 2× slowdown to 30% of the examples on these runs, the median catches it 0–8% of the time. The trimmed mean catches it 88–100% of the time, and it is also steadier between the two runs (8 points apart vs 12).
  • The trade-off, documented: a slowdown confined to fewer than one example in five is treated as noise.
  • Pairs with latency_ms 0 on either side are left out, because 0 is a missing timing.
• Cost is gated on the paired total Σt/Σs − 1, untrimmed. Cost follows tokens, so an expensive answer is a real bill, and trimming would hide a verbose-reasoning blow-up. With no failed calls this is the same number as before (−28.5% / −28.9% on the runs above).
• samples_per_example > 1: each side collapses to one value per example (median latency, mean cost). An errored repeat drops only that sample.
• Cached rows still count, with the latency recorded when they ran live. The policy never excluded them; only the report's run-wide latency figures do. Excluding them here would turn a re-run from cache into "measured nothing", which passes, so re-running a failing suite would clear its gate.
• No interval. A bootstrap rule where a breach counts only when the lower bound clears the budget would almost never fire at n≈21 (run 2's 95% CI is +2.4%..+76.4%). The trimmed estimator is what makes the gate stable. ci_low/ci_high stay null, matching BUNDLE_SPEC.md.
• Signed observed. Savings no longer clip to 0 (−0.289, not 0.0). passed = observed <= allowed stays one-sided. PolicyMetricSummary.cost_increase_rate / latency_increase_rate carry the same signed values.
• denominator counts example pairs (21), not calls on both sides (42).
• Presentation. The report's economics card used to read "Latency +35.6%. Both inside +30% latency". It now adds "judged per example: cost −28.9%, latency +21.6% for the typical example". Insights FACTS get cost_budget_change_pct / latency_budget_change_pct, and the instruction and templated fallback say which figure each ceiling was judged on. The failed-budget table prints increase budgets with their sign.

This changes what two public config keys mean. CHANGELOG.md says so under [Unreleased], and DOCS.md, docs/configuration.md, docs/methodology.md (new section with the evidence) and llms-full.txt are updated.

Hosted bundle

The schema already allows negative observed / *_increase_rate and nullable CIs. The server's re-derive path (app/policy/service.py) compares cost <= allowed, so signed values gate correctly. run_policy runs use the uploaded rows as-is. However, ../evalshift-server/BUNDLE_SPEC.md §"The cost and latency ratios" still describes the old unpaired, clamped mean and the calls-on-both-sides denominator. It needs a matching edit in the server repo; that is not done in this PR.

Verification

• Test-first: new TestPairedCallBudgets in tests/unit/test_policy.py covers spikes not flipping, uniform and >1/5 slowdowns failing, signed savings, one-side-failed pairing, untrimmed cost, samples, errored repeats, the same example_id under two prompts, cached rows, untimed pairs, and the summary values. Existing denominator and zero-valued tests were updated. Facts and report tests were added.
• Re-ran analyze on read-only copies of both runs. Only the two call-derived rows changed. 13d053 goes from fail (the latency row alone) to conditional_pass. 9b78e4 stays inconclusive (its equivalence rate).
• make ci green: ruff, format, mypy --strict, 3418 passed. pre-commit run --all-files green. One pre-existing test, test_cli_capture_remote.py::test_online_sync_says_it_ignores_limits_it_cannot_read, fails locally at narrow terminal widths, on main too. It passes with COLUMNS=300.

🤖 Generated with Claude Code