feat(report): list every regressed example in Top regressions, one card each

CHANGELOG.md

@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **The report's Top regressions lists every example that regressed, one card
+  per example.** It showed the five most negative scored rows, one per example
+  and evaluator. On a real planner run four of the five went to one example
+  that four evaluators each scored −1, so of the ten examples with a drop of
+  0.5 or more it showed two. It now shows every regressed example, worst
+  first, up to 50 per prompt (the per-example breakdown lists all). Each card
+  names every evaluator that dropped the example, gives each one's finding,
+  and shows the outputs, traces and context once. The header counts examples
+  ("15 of 21 examples regressed"), not rows.
+- A drop inside an evaluator's own threshold is no longer listed as a
+  regression. The analysis already tests a `semantic` row above
+  `min_similarity` as no change, yet a 0.93 similarity with a 0.9 threshold
+  sat in the list. Those examples now sit in a closed "N more examples dropped
+  within the evaluator's threshold (scored as no change)" line under the
+  cards, and their cards read "Within threshold" in a neutral tone rather than
+  "Why flagged" in red. `report.json` keeps `top_regressions`, one row per
+  example and evaluator for the examples shown, and adds
+  `regressed_example_count`, `within_threshold_regressions` and
+  `within_threshold_example_count`.
+- The per-example breakdown applies the same rule. It tinted every row with a
+  negative delta, so all 21 rows of that planner run were red under a list of
+  15 regressed examples. Only a regressed example's row is tinted now, it
+  quotes its worst drop beyond a threshold (the delta its card leads with),
+  and a forgiven drop shows muted. Each `example_rows` row in `report.json`
+  carries `regressed`.
+- The report has more room between its sections, between the blocks of a
+  prompt's section, and between each heading and its table.
+
 ## [1.6.0] - 2026-10-22
 
 ### Added

DOCS.md

@@ -226,7 +226,7 @@ init          →   doctor   →   run          →   evaluate       →   analy
 - **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions.
 - **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache.
 - **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict.
-- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. Top regressions are collapsed cards — expand one for the trace diff, the tool diffs and the conversation context. The report is dark-only.
+- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. **Top regressions** lists every example that regressed, worst first, as one collapsed card per example that names each evaluator that dropped it — expand one for each evaluator's finding, then the trace diff, the tool diffs and the conversation context, shown once. It shows at most 50 cards per prompt and says so when it cuts; the per-example breakdown lists every example. A drop inside an evaluator's own threshold (`semantic`'s `min_similarity`) is tested as no change, so it is not a regression: those examples sit in a closed line under the cards, and their cards say "Within threshold" in a neutral tone. The per-example breakdown applies the same rule: only a regressed example's row is tinted, it quotes its worst drop beyond a threshold, and a forgiven drop shows muted. The report is dark-only.
 - **`compare`** chains everything end to end over one suite and adds `--gate`, `--policy-gate`, `--push`, `--open`. Warnings raised while the pipeline runs (LiteLLM deprecation notices, insights-retry notes) are deferred and printed as one `⚠` section directly under the pipeline block — also when a stage fails, since a held-back warning may explain the failure. Errors are never deferred. With `captures.max_age` or `captures.max_cases` set, `compare` prints doctor's `suite freshness` findings for the suite it runs (named with `--suite-name` or auto-selected; not for `--suite <path>`) as `⚠` lines above the verdict, so a CI log shows a stale suite.
 
 ### Artefacts
@@ -240,7 +240,7 @@ Run ids look like `r_20260722_golden_a1b2c3` (`r_<date>_<suite-slug>_<hex>`). Un
 | `scores.jsonl` | evaluate | One line per (pair × evaluator): source/target scores, delta, explanation |
 | `analysis.json` | analyze | Per-comparison statistics, severities, notes |
 | `migration_decision.json` | analyze | Policy verdict + per-budget detail (only when `migration_policy` set) |
-| `report.json` | report | The report payload, including `invariant_violations`: one row per [trace rule](#trace-invariants) the target broke |
+| `report.json` | report | The report payload, including `invariant_violations`: one row per [trace rule](#trace-invariants) the target broke. Each prompt section's `top_regressions` holds one row per example × evaluator for the regressed examples shown (at most 50), and `regressed_example_count` counts every regressed example. `within_threshold_regressions` / `within_threshold_example_count` do the same for examples whose *every* drop stayed inside its evaluator's threshold; a forgiven drop on an example that also regressed is in neither list. Each `example_rows` row carries `regressed` |
 | `report.html` | report | Single-file HTML report |
 | `insights.json` | report | Cached [run insights](#run-insights) narrative (optional; skipped with `--no-insights` or no API key) |
 | `traces.jsonl` | traces import | Bring-your-own agent traces (optional) |

README.md

@@ -303,8 +303,8 @@ assets, works offline) has:
   behavioural drift explained by `defaults.insights_model`. Every figure in it
   is copied from the computed statistics, never generated. Needs a provider
   key; skip it with `--no-insights`.
-* **Per-prompt deep dive** — aggregate stats, per-slice breakdown,
-  top-5 worst regressions side-by-side.
+* **Per-prompt deep dive** — aggregate stats, per-slice breakdown, and
+  every regressed example side-by-side, worst first, one card per example.
 * **Methodology appendix** — every test, p-value, effect size, and
   CI is documented.
 

docs/agents.md

@@ -602,8 +602,8 @@ Two places name the tools themselves, both read off the evaluator's own
 record rather than re-derived from the call trace:
 
 * the **per-example breakdown** gains a `Tools called (source → target)`
-  column — one line per example, so a ten-example suite shows all ten
-  rather than only the five worst;
+  column — one line per example, so a ten-example suite shows all ten,
+  regressed or not;
 * each **top regression** card states both sides' tools in its "why
   flagged" line.
 

llms-full.txt

@@ -96,7 +96,7 @@ as one section under the pipeline block (also on stage failure); errors are neve
 | scores.jsonl | evaluate | one EvalRecord per (pair x evaluator): source/target scores in [0,1], delta |
 | analysis.json | analyze | per-(prompt,evaluator,slice) statistics + severity |
 | migration_decision.json | analyze | policy verdict (only when migration_policy configured) |
-| report.json + report.html | report | payload + single-file HTML (no external assets, dark-only). report.json top-level invariant_violations: one row per trace rule the target broke (see evaluators.trace_invariants) |
+| report.json + report.html | report | payload + single-file HTML (no external assets, dark-only). report.json top-level invariant_violations: one row per trace rule the target broke (see evaluators.trace_invariants). Per prompt section: top_regressions (one row per example x evaluator, regressed examples shown, at most 50), regressed_example_count (all of them), within_threshold_regressions + within_threshold_example_count (examples whose every drop stayed inside its evaluator's threshold; a forgiven drop on a regressed example is in neither), example_rows[].regressed |
 | insights.json | report | cached machine-written narrative (optional; --no-insights skips) |
 | traces.jsonl | traces import | optional BYO agent traces |
 | run_bundle.json.gz | bundle/push | optional hosted upload bundle |
@@ -253,7 +253,13 @@ evalshift report RUN_ID [-c CONFIG] [--open] [--insights/--no-insights (default
   evaluators; advisory ones only on a prompt nothing else scored) -> insights narrative ->
   "Trace rules broken" panel (only when the target broke a hand-written trace rule;
   blocking: false rows tagged advisory) -> one section per prompt (economics, per-example,
-  by-evaluator, slices, top regressions as collapsible cards) -> methodology. Dark-only, no external assets, no JS. Rendered surfaces show
+  by-evaluator, slices, top regressions) -> methodology. Top regressions: one collapsed card
+  per regressed example, worst first, naming every evaluator that dropped it; at most 50 per
+  prompt (the per-example table lists all); a drop inside an evaluator's own threshold
+  (semantic min_similarity) is tested as no change, so it sits in a closed line below the
+  cards ("Within threshold", neutral tone), not among them; the per-example table tints
+  only regressed rows and shows a forgiven drop muted. Dark-only, no external assets, no
+  JS. Rendered surfaces show
   display names, never internal identifiers: failure categories as plain language
   (TOOL_SELECTION_DRIFT -> "Different tools chosen"; map in evaluators/failures.py), budgets
   as their labels (max_tool_divergence -> "Tool-selection divergence"; map in

src/evalshift_cli/insights/facts.py

@@ -117,7 +117,7 @@ class ExampleFact:
 
     Deliberately not ``reports.json.ExampleRow`` — that row carries the deltas
     but no output text, and ``TopRegression`` carries the text but only for the
-    worst five of each prompt. The insights path needs both on every example.
+    examples that regressed. The insights path needs both on every example.
     """
 
     example_id: str

src/evalshift_cli/reports/html.py

@@ -333,7 +333,7 @@ def _kind_label(kind: str) -> str:
     return f"{label} — {axis}" if axis else label
 
 
-def _regression_reason(regression: TopRegression) -> str:
+def _regression_reason(regression: TopRegression, within: bool = False) -> str:
     """Explain, in one plain sentence, why an example was flagged.
 
     Prefers the evaluator's own written rationale (e.g. an llm_judge verdict).
@@ -345,7 +345,24 @@ def _regression_reason(regression: TopRegression) -> str:
     user's: on the run this work came from both tool axes were called
     ``routing``, which matched no family prefix, so every tool regression
     rendered its scores and no reason at all.
+
+    Args:
+        regression: The drop to explain.
+        within: The drop is inside its evaluator's own threshold, so the
+            analysis tests it as no change. The sentence says that instead
+            of claiming the meaning drifted.
     """
+    if within:
+        raw = regression.metadata.get("raw_cosine")
+        similar = (
+            f" ({raw * 100:.0f}% similar; 100% = identical)"
+            if isinstance(raw, (int, float))
+            else ""
+        )
+        return (
+            f"The target reworded the answer but stayed inside the evaluator's threshold"
+            f"{similar}, so the analysis scores it as no change."
+        )
     text = regression.explanation.strip()
     if text:
         return text