feat(report): fold long tables after ten rows under a "Show N more" label

CHANGELOG.md

@@ -34,6 +34,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   quotes its worst drop beyond a threshold (the delta its card leads with),
   and a forgiven drop shows muted. Each `example_rows` row in `report.json`
   carries `regressed`.
+- A regression card outlines the target's trace in red only when its tool
+  calls differ from the source's, and reads a target's "(no tool calls)" in
+  red only when the source did call tools. A card flagged for its wording
+  alone showed both sides making the same call with the target's in a red
+  box, which read as a tool failure.
+- The per-example breakdown and the Trace rules broken table show their
+  first 10 rows; the rest unfold under a "Show N more examples" / "Show N
+  more broken rules" label, in the same table so the columns stay aligned.
+  No script: a checkbox drives it, so it works offline and from an email
+  attachment, and a printed report shows every row.
 - The report has more room between its sections, between the blocks of a
   prompt's section, and between each heading and its table.
 

DOCS.md

@@ -226,7 +226,7 @@ init          →   doctor   →   run          →   evaluate       →   analy
 - **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions.
 - **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache.
 - **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict.
-- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. **Top regressions** lists every example that regressed, worst first, as one collapsed card per example that names each evaluator that dropped it — expand one for each evaluator's finding, then the trace diff, the tool diffs and the conversation context, shown once. It shows at most 50 cards per prompt and says so when it cuts; the per-example breakdown lists every example. A drop inside an evaluator's own threshold (`semantic`'s `min_similarity`) is tested as no change, so it is not a regression: those examples sit in a closed line under the cards, and their cards say "Within threshold" in a neutral tone. The per-example breakdown applies the same rule: only a regressed example's row is tinted, it quotes its worst drop beyond a threshold, and a forgiven drop shows muted. The report is dark-only.
+- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. **Top regressions** lists every example that regressed, worst first, as one collapsed card per example that names each evaluator that dropped it — expand one for each evaluator's finding, then the trace diff, the tool diffs and the conversation context, shown once. It shows at most 50 cards per prompt and says so when it cuts; the per-example breakdown lists every example. A drop inside an evaluator's own threshold (`semantic`'s `min_similarity`) is tested as no change, so it is not a regression: those examples sit in a closed line under the cards, and their cards say "Within threshold" in a neutral tone. The per-example breakdown applies the same rule: only a regressed example's row is tinted, it quotes its worst drop beyond a threshold, and a forgiven drop shows muted. It and the **Trace rules broken** table show their first 10 rows and fold the rest under a "Show N more" label (no script; a printed report shows every row). The report is dark-only.
 - **`compare`** chains everything end to end over one suite and adds `--gate`, `--policy-gate`, `--push`, `--open`. Warnings raised while the pipeline runs (LiteLLM deprecation notices, insights-retry notes) are deferred and printed as one `⚠` section directly under the pipeline block — also when a stage fails, since a held-back warning may explain the failure. Errors are never deferred. With `captures.max_age` or `captures.max_cases` set, `compare` prints doctor's `suite freshness` findings for the suite it runs (named with `--suite-name` or auto-selected; not for `--suite <path>`) as `⚠` lines above the verdict, so a CI log shows a stale suite.
 
 ### Artefacts

llms-full.txt

@@ -258,7 +258,8 @@ evalshift report RUN_ID [-c CONFIG] [--open] [--insights/--no-insights (default
   prompt (the per-example table lists all); a drop inside an evaluator's own threshold
   (semantic min_similarity) is tested as no change, so it sits in a closed line below the
   cards ("Within threshold", neutral tone), not among them; the per-example table tints
-  only regressed rows and shows a forgiven drop muted. Dark-only, no external assets, no
+  only regressed rows and shows a forgiven drop muted. The per-example and trace-rules
+  tables show 10 rows and fold the rest under "Show N more" (CSS only; print shows all). Dark-only, no external assets, no
   JS. Rendered surfaces show
   display names, never internal identifiers: failure categories as plain language
   (TOOL_SELECTION_DRIFT -> "Different tools chosen"; map in evaluators/failures.py), budgets

src/evalshift_cli/reports/html.py

@@ -36,6 +36,10 @@
 
 REPORT_HTML_FILENAME: str = "report.html"
 
+#: Rows a long table (the per-example breakdown, the trace rules broken) shows
+#: before its "Show N more" toggle. The rest are in the page, folded, not cut.
+TABLE_ROWS_SHOWN: int = 10
+
 _TEMPLATE_DIR = Path(__file__).parent / "templates"
 
 # Icon per ``InsightFinding.kind``. Matches the hosted design's ✓ / ✕ / !.
@@ -599,6 +603,7 @@ def render_html(report: ReportData, *, insight: Insight | None = None) -> str:
     env.globals["confidence_label"] = _confidence_label
     env.globals["effect_magnitude"] = _effect_magnitude
     env.globals["small_sample_threshold"] = _SMALL_SAMPLE_THRESHOLD
+    env.globals["rows_shown"] = TABLE_ROWS_SHOWN
     env.globals["latency"] = _latency
     env.globals["evaluator_label"] = _evaluator_label
     env.globals["axis_of"] = _axis_of
@@ -666,4 +671,4 @@ def write_html(report: ReportData, run_dir: Path, *, insight: Insight | None = N
     return out_path
 
 
-__all__ = ["REPORT_HTML_FILENAME", "render_html", "write_html"]
+__all__ = ["REPORT_HTML_FILENAME", "TABLE_ROWS_SHOWN", "render_html", "write_html"]

src/evalshift_cli/reports/templates/report.css

@@ -154,6 +154,8 @@ pre {
 }
 .notice p { margin: 0; font-size: 14.5px; line-height: 1.65; color: var(--body); }
 .notice-bad { background: var(--bad-tint); border-color: var(--bad-edge); }
+/* A finding that is not a problem (a drop inside its own threshold). */
+.notice-neutral { background: var(--well); border-color: var(--stroke); }
 
 /* --- Header ------------------------------------------------------------- */
 
@@ -264,6 +266,27 @@ h4 {
 /* Blocks inside one section are separate findings: give each its own room
    rather than butting a heading against the table above it. */
 .block + .block { margin-top: 20px; }
+
+/* A long table folds its rows past the first ten under a label wired to a
+   visually hidden checkbox: no script, and the checkbox keeps keyboard focus. */
+.more-toggle { position: absolute; width: 1px; height: 1px; opacity: 0; pointer-events: none; }
+.more-toggle:not(:checked) ~ .panel .row-more { display: none; }
+.more-label {
+  display: inline-flex;
+  align-items: center;
+  gap: 10px;
+  align-self: flex-start;
+  padding: 4px 0;
+  font-family: var(--font-mono);
+  font-size: 13px;
+  color: var(--accent);
+  cursor: pointer;
+}
+.more-label:hover { color: var(--fg); }
+.more-toggle:focus-visible ~ .more-label { outline: 2px solid var(--accent); outline-offset: 2px; }
+.more-toggle:checked ~ .more-label .caret::after { content: "\25BE"; }
+.more-toggle:checked ~ .more-label .more-show,
+.more-toggle:not(:checked) ~ .more-label .more-hide { display: none; }
 p.empty {
   margin: 0;
   padding: 22px;
@@ -831,6 +854,9 @@ footer p { margin: 0; }
      engines with ::details-content (Chromium 131+) honour it. */
   .regression::details-content { content-visibility: visible; }
   .reg-toggle { display: none; }
+  /* Folded table rows print too: a printed report must not lose examples. */
+  .row-more { display: table-row !important; }
+  .more-label { display: none; }
 }
 
 /* --- Tone utilities ------------------------------------------------------ */

src/evalshift_cli/reports/templates/report.html.j2

@@ -34,6 +34,20 @@
   {%- endif -%}
 {%- endmacro -%}
 
+{#- A long table shows its first ``rows_shown`` rows; the rest unfold under a
+    "Show N more" label. A checkbox and a label rather than a script, since
+    the report runs no JS, and one table rather than two, so the columns stay
+    aligned. The checkbox goes before the table's panel, the label after it. -#}
+{%- macro more_toggle(id, total) -%}
+  {%- if total > rows_shown %}<input type="checkbox" class="more-toggle" id="{{ id }}">{% endif -%}
+{%- endmacro -%}
+{%- macro more_label(id, total, noun, plural) -%}
+  {%- if total > rows_shown -%}
+  {%- set hidden = total - rows_shown -%}
+  <label class="more-label" for="{{ id }}"><span class="caret"></span><span class="more-show">Show {{ hidden }} more {{ noun if hidden == 1 else plural }}</span><span class="more-hide">Show fewer</span></label>
+  {%- endif -%}
+{%- endmacro -%}
+
 {#- One regressed example: why each evaluator flagged it, then the pair's
     outputs and context once. The outputs are the same whichever evaluator
     scored them, so a card per evaluator repeated them and let one example
@@ -44,6 +58,11 @@
 {%- set target_tone = "muted" if within else "bad" -%}
 {%- set first = card.regressions[0] -%}
 {%- set single = card.regressions|length == 1 -%}
+{#- The target's trace reads red only where its calls differ from the
+    source's: a matching trace under a red box read as a tool failure on
+    cards flagged for their wording alone. -#}
+{%- set agent_differs = first.trace_diff is not none and first.trace_diff.items|length > 0 -%}
+{%- set tools_differ = first.tool_diffs|length > 0 -%}
 <details class="panel regression"{% if opened %} open{% endif %}>
   <summary class="reg-head">
     <span class="caret"></span>
@@ -58,7 +77,7 @@
   <div class="reg-body">
     {% for tr in card.regressions %}
     {% set reason = regression_reason(tr, within) %}
-    <div class="notice{{ " notice-bad" if not within }} reg-why">
+    <div class="notice {{ "notice-neutral" if within else "notice-bad" }} reg-why">
       <div class="reg-why-line">
         <span class="eyebrow">{{ "Within threshold" if within else "Why flagged" }}</span>
         {% if not single %}<span class="reg-why-evaluator">{{ evaluator_label(tr.evaluator_name, tr.kind) }}</span>{% endif %}
@@ -100,13 +119,13 @@
         <div class="reg-col-head"><span class="eyebrow">Target agent trace</span>{% if single %}<span class="reg-col-score {{ target_tone }}">score {{ "%.3f"|format(first.target_score) }}</span>{% endif %}</div>
         {% if first.target_agent_trace and first.target_agent_trace.tool_calls %}
         {% set source_names = (first.source_agent_trace.tool_calls if first.source_agent_trace else []) | map(attribute='name') | list %}
-        <ol class="trace-list trace-target">
+        <ol class="trace-list{{ ' trace-target' if agent_differs }}">
           {% for c in first.target_agent_trace.tool_calls %}
           <li class="trace-call {{ "" if c.name in source_names else "trace-extra" }}">{{ c.name }} <span class="trace-args">({{ c.arguments|tojson }})</span></li>
           {% endfor %}
         </ol>
         {% else %}
-        <p class="trace-empty trace-missing">(no tool calls)</p>
+        <p class="trace-empty{{ ' trace-missing' if agent_differs }}">(no tool calls)</p>
         {% endif %}
       </div>
     </div>
@@ -149,7 +168,7 @@
         <div class="reg-col-head"><span class="eyebrow">Target trace</span>{% if single %}<span class="reg-col-score {{ target_tone }}">score {{ "%.3f"|format(first.target_score) }}</span>{% endif %}</div>
         {% if first.target_trace and first.target_trace.calls %}
         {% set source_names = (first.source_trace.calls if first.source_trace else []) | map(attribute='tool_name') | list %}
-        <ol class="trace-list trace-target">
+        <ol class="trace-list{{ ' trace-target' if tools_differ }}">
           {% for round_index, calls in first.target_trace.calls|groupby("round_index") %}
           {% if first.target_trace.round_count > 1 %}<li class="trace-round-label" value="0">round {{ round_index + 1 }}</li>{% endif %}
           {% for c in calls %}
@@ -158,7 +177,7 @@
           {% endfor %}
         </ol>
         {% else %}
-        <p class="trace-empty trace-missing">(no tool calls)</p>
+        <p class="trace-empty{{ ' trace-missing' if tools_differ }}">(no tool calls)</p>
         {% endif %}
         {% if first.target_trace and first.target_trace.final_text %}
         <p class="trace-final"><em>final text:</em> {{ first.target_trace.final_text }}</p>
@@ -590,12 +609,13 @@
         <span class="mono muted count-note">{{ invariant_violations|length }} by the target</span>
       </div>
       <p class="sub">Every rule the target broke, whatever the score change. A rule both models broke scores no change, so it never shows among the regressions below.</p>
+      {{ more_toggle("more-violations", invariant_violations|length) }}
       <div class="panel scroll">
         <table>
           <thead><tr><th>Example</th><th>Rule</th><th>Owner</th><th>What happened</th><th>Source too?</th></tr></thead>
           <tbody>
           {% for v in invariant_violations %}
-            <tr>
+            <tr{% if loop.index > rows_shown %} class="row-more"{% endif %}>
               <td class="mono">{{ v.prompt_id }} · {{ v.example_id }}{% if v.round_index %} · round {{ v.round_index + 1 }}{% endif %}{% if v.sample is not none %} · sample {{ v.sample + 1 }}{% endif %}</td>
               <td><span class="badge badge-bad">{{ v.rule_id }}</span> <span class="muted">{{ v.rule_type }}</span>{% if not v.blocking %} <span class="badge advisory-tag" title="blocking: false in evalshift.yaml — reported, never gates">advisory</span>{% endif %}</td>
               <td>{{ v.owner or "—" }}</td>
@@ -606,6 +626,7 @@
           </tbody>
         </table>
       </div>
+      {{ more_label("more-violations", invariant_violations|length, "broken rule", "broken rules") }}
     </div>
   </section>
 {% endif %}
@@ -663,6 +684,8 @@
       {% if section.example_rows %}
       {% set worst = section.example_rows|map(attribute="worst_delta_score")|reject("none")|map("abs")|list %}
       {% set scale = worst|max if worst else 0 %}
+      {% set more_id = "more-examples-" ~ loop.index %}
+      {{ more_toggle(more_id, section.example_rows|length) }}
       <div class="panel scroll">
         <table class="examples">
           <thead><tr>
@@ -676,7 +699,7 @@
           </tr></thead>
           <tbody>
           {% for row in section.example_rows %}
-            <tr class="{% if row.regressed %}ex-regression{% elif row.worst_delta_score is not none and row.worst_delta_score > 0 %}ex-improved{% endif %}">
+            <tr class="{% if row.regressed %}ex-regression{% elif row.worst_delta_score is not none and row.worst_delta_score > 0 %}ex-improved{% endif %}{% if loop.index > rows_shown %} row-more{% endif %}">
               <td class="num" style="text-align:left"><code>{{ row.example_id }}</code>{% if row.turn_index is not none %}<span class="turn-badge">turn {{ row.turn_index }}</span>{% endif %}</td>
               {# One dot-separated line, not a chip per tag: a stack of chips in a
                  narrow column pushes every other row taller. #}
@@ -705,6 +728,7 @@
           </tbody>
         </table>
       </div>
+      {{ more_label(more_id, section.example_rows|length, "example", "examples") }}
       {% else %}
       <div class="panel"><p class="empty">No examples available.</p></div>
       {% endif %}