[babaliauskas]

What this changes

Top regressions lists every example that regressed, one card per example.

• Until now it showed the five most negative scored rows, one per example × evaluator.
  • On the real testApp planner run (21 examples), four of the five slots went to one example that four evaluators each scored −1.
  • So of the ten examples with a drop of 0.5 or more, it showed two. The two LLM-judge losses and both broken agent contracts were missing.
• Now:
  • every regressed example is shown, worst first, up to MAX_REGRESSION_CARDS = 50 per prompt;
  • each card names every evaluator that dropped the example and gives each one's finding;
  • the outputs, traces and context show once per card.
• The header counts examples, not rows: "15 of 21 examples regressed".
• Past the cap, the header says it is showing the worst 50 and points to the per-example breakdown.

A drop inside an evaluator's own threshold is not listed as a regression.

• analyze already tests a semantic row above min_similarity as no change (analysis.tolerance.within_tolerance). Yet the list showed a 0.93 similarity, with a 0.9 threshold, as a regression.
• The list now uses the same rule. Those examples sit in a closed line under the cards: "N more examples dropped within the evaluator's threshold (scored as no change)".
• Their cards read "Within threshold" in a neutral tone, not "Why flagged" in red.

The per-example breakdown uses the same rule.

• It used to tint every row with a negative delta, so all 21 planner rows were red.
• Now only a regressed example's row is tinted, and it shows its worst drop beyond a threshold. A forgiven drop shows muted.

More spacing.

• Between sections: 30px → 56px.
• Between the blocks of a prompt's section: 20px → 40px.
• Between each heading and its table: 10px → 14px.
• Mobile gets smaller values.

report.json.

• top_regressions keeps its per-row shape: one row per example × evaluator, for the examples shown.
• New fields: regressed_example_count, within_threshold_regressions, within_threshold_example_count, and example_rows[].regressed.
• The push bundle doesn't carry report.json, and the hosted server builds its own regression list, so nothing hosted changes.

Results on real runs

Run	Before	After
planner (21 examples)	5 rows, 2 examples	15 cards; 6 within-threshold examples in the closed line
converter (12 examples)	5 rows, 2 of them within threshold	3 cards; 7 within-threshold examples

Review

An independent review approved this with fixes, all now in the second commit:

• within-threshold cards still claimed a regression;
• the breakdown's tint contradicted the count;
• tie-order, within-list-cap and empty-list tests were missing;
• two comments were stale;
• the report.json doc wording was imprecise.

Known trade-off: the report grows with the number of regressed examples. On the planner run, report.html went from 115 KB to 261 KB.

Not changed here: the narrative's regression samples (insights/stage.py) still pick examples by raw worst delta. So the narrative can still describe a within-threshold drop as a regression. That's a separate follow-up.

Test plan

• make ci: ruff, ruff format, mypy --strict, and pytest (3393 passed, 95.3% coverage)
• pre-commit run --all-files
• New tests in TestTopRegressionsListEveryRealRegression, each written first and seen to fail:
  • threshold exclusion;
  • one card per example, worst first;
  • errored and positive rows left out;
  • ordering and ties;
  • both caps;
  • header text and the empty state;
  • within-card wording;
  • breakdown tint and delta;
  • report.json fields.
• Rendered the real planner and converter runs and checked headless-Chromium screenshots of the cards, the open within-threshold list and the breakdown

🤖 Generated with Claude Code