diff --git a/CHANGELOG.md b/CHANGELOG.md index c1a3ad1..b0c0d2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 quotes its worst drop beyond a threshold (the delta its card leads with), and a forgiven drop shows muted. Each `example_rows` row in `report.json` carries `regressed`. +- A regression card outlines the target's trace in red only when its tool + calls differ from the source's, and reads a target's "(no tool calls)" in + red only when the source did call tools. A card flagged for its wording + alone showed both sides making the same call with the target's in a red + box, which read as a tool failure. +- The per-example breakdown and the Trace rules broken table show their + first 10 rows; the rest unfold under a "Show N more examples" / "Show N + more broken rules" label, in the same table so the columns stay aligned. + No script: a checkbox drives it, so it works offline and from an email + attachment, and a printed report shows every row. - The report has more room between its sections, between the blocks of a prompt's section, and between each heading and its table. diff --git a/DOCS.md b/DOCS.md index 73765a1..6d1bdbc 100644 --- a/DOCS.md +++ b/DOCS.md @@ -226,7 +226,7 @@ init → doctor → run → evaluate → analy - **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions. - **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache. - **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict. -- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. **Top regressions** lists every example that regressed, worst first, as one collapsed card per example that names each evaluator that dropped it — expand one for each evaluator's finding, then the trace diff, the tool diffs and the conversation context, shown once. It shows at most 50 cards per prompt and says so when it cuts; the per-example breakdown lists every example. A drop inside an evaluator's own threshold (`semantic`'s `min_similarity`) is tested as no change, so it is not a regression: those examples sit in a closed line under the cards, and their cards say "Within threshold" in a neutral tone. The per-example breakdown applies the same rule: only a regressed example's row is tinted, it quotes its worst drop beyond a threshold, and a forgiven drop shows muted. The report is dark-only. +- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. **Top regressions** lists every example that regressed, worst first, as one collapsed card per example that names each evaluator that dropped it — expand one for each evaluator's finding, then the trace diff, the tool diffs and the conversation context, shown once. It shows at most 50 cards per prompt and says so when it cuts; the per-example breakdown lists every example. A drop inside an evaluator's own threshold (`semantic`'s `min_similarity`) is tested as no change, so it is not a regression: those examples sit in a closed line under the cards, and their cards say "Within threshold" in a neutral tone. The per-example breakdown applies the same rule: only a regressed example's row is tinted, it quotes its worst drop beyond a threshold, and a forgiven drop shows muted. It and the **Trace rules broken** table show their first 10 rows and fold the rest under a "Show N more" label (no script; a printed report shows every row). The report is dark-only. - **`compare`** chains everything end to end over one suite and adds `--gate`, `--policy-gate`, `--push`, `--open`. Warnings raised while the pipeline runs (LiteLLM deprecation notices, insights-retry notes) are deferred and printed as one `⚠` section directly under the pipeline block — also when a stage fails, since a held-back warning may explain the failure. Errors are never deferred. With `captures.max_age` or `captures.max_cases` set, `compare` prints doctor's `suite freshness` findings for the suite it runs (named with `--suite-name` or auto-selected; not for `--suite `) as `⚠` lines above the verdict, so a CI log shows a stale suite. ### Artefacts diff --git a/llms-full.txt b/llms-full.txt index 50fb763..e709a1b 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -258,7 +258,8 @@ evalshift report RUN_ID [-c CONFIG] [--open] [--insights/--no-insights (default prompt (the per-example table lists all); a drop inside an evaluator's own threshold (semantic min_similarity) is tested as no change, so it sits in a closed line below the cards ("Within threshold", neutral tone), not among them; the per-example table tints - only regressed rows and shows a forgiven drop muted. Dark-only, no external assets, no + only regressed rows and shows a forgiven drop muted. The per-example and trace-rules + tables show 10 rows and fold the rest under "Show N more" (CSS only; print shows all). Dark-only, no external assets, no JS. Rendered surfaces show display names, never internal identifiers: failure categories as plain language (TOOL_SELECTION_DRIFT -> "Different tools chosen"; map in evaluators/failures.py), budgets diff --git a/src/evalshift_cli/reports/html.py b/src/evalshift_cli/reports/html.py index 8c117bd..bb7115e 100644 --- a/src/evalshift_cli/reports/html.py +++ b/src/evalshift_cli/reports/html.py @@ -36,6 +36,10 @@ REPORT_HTML_FILENAME: str = "report.html" +#: Rows a long table (the per-example breakdown, the trace rules broken) shows +#: before its "Show N more" toggle. The rest are in the page, folded, not cut. +TABLE_ROWS_SHOWN: int = 10 + _TEMPLATE_DIR = Path(__file__).parent / "templates" # Icon per ``InsightFinding.kind``. Matches the hosted design's ✓ / ✕ / !. @@ -599,6 +603,7 @@ def render_html(report: ReportData, *, insight: Insight | None = None) -> str: env.globals["confidence_label"] = _confidence_label env.globals["effect_magnitude"] = _effect_magnitude env.globals["small_sample_threshold"] = _SMALL_SAMPLE_THRESHOLD + env.globals["rows_shown"] = TABLE_ROWS_SHOWN env.globals["latency"] = _latency env.globals["evaluator_label"] = _evaluator_label env.globals["axis_of"] = _axis_of @@ -666,4 +671,4 @@ def write_html(report: ReportData, run_dir: Path, *, insight: Insight | None = N return out_path -__all__ = ["REPORT_HTML_FILENAME", "render_html", "write_html"] +__all__ = ["REPORT_HTML_FILENAME", "TABLE_ROWS_SHOWN", "render_html", "write_html"] diff --git a/src/evalshift_cli/reports/templates/report.css b/src/evalshift_cli/reports/templates/report.css index 722d94d..9ba3c43 100644 --- a/src/evalshift_cli/reports/templates/report.css +++ b/src/evalshift_cli/reports/templates/report.css @@ -154,6 +154,8 @@ pre { } .notice p { margin: 0; font-size: 14.5px; line-height: 1.65; color: var(--body); } .notice-bad { background: var(--bad-tint); border-color: var(--bad-edge); } +/* A finding that is not a problem (a drop inside its own threshold). */ +.notice-neutral { background: var(--well); border-color: var(--stroke); } /* --- Header ------------------------------------------------------------- */ @@ -264,6 +266,27 @@ h4 { /* Blocks inside one section are separate findings: give each its own room rather than butting a heading against the table above it. */ .block + .block { margin-top: 20px; } + +/* A long table folds its rows past the first ten under a label wired to a + visually hidden checkbox: no script, and the checkbox keeps keyboard focus. */ +.more-toggle { position: absolute; width: 1px; height: 1px; opacity: 0; pointer-events: none; } +.more-toggle:not(:checked) ~ .panel .row-more { display: none; } +.more-label { + display: inline-flex; + align-items: center; + gap: 10px; + align-self: flex-start; + padding: 4px 0; + font-family: var(--font-mono); + font-size: 13px; + color: var(--accent); + cursor: pointer; +} +.more-label:hover { color: var(--fg); } +.more-toggle:focus-visible ~ .more-label { outline: 2px solid var(--accent); outline-offset: 2px; } +.more-toggle:checked ~ .more-label .caret::after { content: "\25BE"; } +.more-toggle:checked ~ .more-label .more-show, +.more-toggle:not(:checked) ~ .more-label .more-hide { display: none; } p.empty { margin: 0; padding: 22px; @@ -831,6 +854,9 @@ footer p { margin: 0; } engines with ::details-content (Chromium 131+) honour it. */ .regression::details-content { content-visibility: visible; } .reg-toggle { display: none; } + /* Folded table rows print too: a printed report must not lose examples. */ + .row-more { display: table-row !important; } + .more-label { display: none; } } /* --- Tone utilities ------------------------------------------------------ */ diff --git a/src/evalshift_cli/reports/templates/report.html.j2 b/src/evalshift_cli/reports/templates/report.html.j2 index f016662..81c903b 100644 --- a/src/evalshift_cli/reports/templates/report.html.j2 +++ b/src/evalshift_cli/reports/templates/report.html.j2 @@ -34,6 +34,20 @@ {%- endif -%} {%- endmacro -%} +{#- A long table shows its first ``rows_shown`` rows; the rest unfold under a + "Show N more" label. A checkbox and a label rather than a script, since + the report runs no JS, and one table rather than two, so the columns stay + aligned. The checkbox goes before the table's panel, the label after it. -#} +{%- macro more_toggle(id, total) -%} + {%- if total > rows_shown %}{% endif -%} +{%- endmacro -%} +{%- macro more_label(id, total, noun, plural) -%} + {%- if total > rows_shown -%} + {%- set hidden = total - rows_shown -%} + + {%- endif -%} +{%- endmacro -%} + {#- One regressed example: why each evaluator flagged it, then the pair's outputs and context once. The outputs are the same whichever evaluator scored them, so a card per evaluator repeated them and let one example @@ -44,6 +58,11 @@ {%- set target_tone = "muted" if within else "bad" -%} {%- set first = card.regressions[0] -%} {%- set single = card.regressions|length == 1 -%} +{#- The target's trace reads red only where its calls differ from the + source's: a matching trace under a red box read as a tool failure on + cards flagged for their wording alone. -#} +{%- set agent_differs = first.trace_diff is not none and first.trace_diff.items|length > 0 -%} +{%- set tools_differ = first.tool_diffs|length > 0 -%}
@@ -58,7 +77,7 @@
{% for tr in card.regressions %} {% set reason = regression_reason(tr, within) %} -
+
{{ "Within threshold" if within else "Why flagged" }} {% if not single %}{{ evaluator_label(tr.evaluator_name, tr.kind) }}{% endif %} @@ -100,13 +119,13 @@
Target agent trace{% if single %}score {{ "%.3f"|format(first.target_score) }}{% endif %}
{% if first.target_agent_trace and first.target_agent_trace.tool_calls %} {% set source_names = (first.source_agent_trace.tool_calls if first.source_agent_trace else []) | map(attribute='name') | list %} -
    +
      {% for c in first.target_agent_trace.tool_calls %}
    1. {{ c.name }} ({{ c.arguments|tojson }})
      2. {% endfor %}
    {% else %} -

    (no tool calls)

    +

    (no tool calls)

    {% endif %}
@@ -149,7 +168,7 @@
Target trace{% if single %}score {{ "%.3f"|format(first.target_score) }}{% endif %}
{% if first.target_trace and first.target_trace.calls %} {% set source_names = (first.source_trace.calls if first.source_trace else []) | map(attribute='tool_name') | list %} -
    +
      {% for round_index, calls in first.target_trace.calls|groupby("round_index") %} {% if first.target_trace.round_count > 1 %}
    1. round {{ round_index + 1 }}
      2. {% endif %} {% for c in calls %} @@ -158,7 +177,7 @@ {% endfor %}
    {% else %} -

    (no tool calls)

    +

    (no tool calls)

    {% endif %} {% if first.target_trace and first.target_trace.final_text %}

    final text: {{ first.target_trace.final_text }}

    @@ -590,12 +609,13 @@ {{ invariant_violations|length }} by the target

Every rule the target broke, whatever the score change. A rule both models broke scores no change, so it never shows among the regressions below.

+ {{ more_toggle("more-violations", invariant_violations|length) }}
{% for v in invariant_violations %} - + rows_shown %} class="row-more"{% endif %}> @@ -606,6 +626,7 @@
ExampleRuleOwnerWhat happenedSource too?
{{ v.prompt_id }} · {{ v.example_id }}{% if v.round_index %} · round {{ v.round_index + 1 }}{% endif %}{% if v.sample is not none %} · sample {{ v.sample + 1 }}{% endif %} {{ v.rule_id }} {{ v.rule_type }}{% if not v.blocking %} advisory{% endif %} {{ v.owner or "—" }}
+ {{ more_label("more-violations", invariant_violations|length, "broken rule", "broken rules") }}
{% endif %} @@ -663,6 +684,8 @@ {% if section.example_rows %} {% set worst = section.example_rows|map(attribute="worst_delta_score")|reject("none")|map("abs")|list %} {% set scale = worst|max if worst else 0 %} + {% set more_id = "more-examples-" ~ loop.index %} + {{ more_toggle(more_id, section.example_rows|length) }}
@@ -676,7 +699,7 @@ {% for row in section.example_rows %} - + {# One dot-separated line, not a chip per tag: a stack of chips in a narrow column pushes every other row taller. #} @@ -705,6 +728,7 @@
{{ row.example_id }}{% if row.turn_index is not none %}turn {{ row.turn_index }}{% endif %}
+ {{ more_label(more_id, section.example_rows|length, "example", "examples") }} {% else %}

No examples available.

{% endif %} diff --git a/tests/unit/test_reports.py b/tests/unit/test_reports.py index 320c1f8..d3fa2e3 100644 --- a/tests/unit/test_reports.py +++ b/tests/unit/test_reports.py @@ -3180,6 +3180,60 @@ def test_a_regressed_rows_worst_delta_is_the_regression(self, tmp_path: Path) -> assert (row.regressed, row.worst_delta_score) == (True, -0.05) assert section.regressed_examples[0].regressions[0].delta == -0.05 + def _traced_run( + self, tmp_path: Path, *, source_tools: list[str], target_tools: list[str] + ) -> Path: + """ex1 regressed on semantic, with each side's recorded tool calls.""" + tmp_path.mkdir(parents=True, exist_ok=True) + run_dir = self._run(tmp_path, [self._semantic("ex1", -0.2, beyond_threshold=True)]) + (run_dir / "raw.jsonl").unlink() + for role, tools in (("source", source_tools), ("target", target_tools)): + calls = [ + ToolCall(tool_name=name, arguments={"value": 5}, sequence_index=i) + for i, name in enumerate(tools) + ] + append_call( + run_dir, + Call( + run_id="r_20260601_aaaaaa", + prompt_id="greet", + example_id="ex1", + model_id=f"model-{role}", + role=role, + text="5 kg is 11.02 lb.", + cost_usd=0.0001, + latency_ms=100, + input_tokens=10, + output_tokens=5, + trace=ToolTrace(calls=calls), + ), + ) + return run_dir + + def test_the_target_trace_is_red_only_when_its_calls_differ(self, tmp_path: Path) -> None: + same = self._traced_run( + tmp_path / "same", source_tools=["convert_units"], target_tools=["convert_units"] + ) + differ = self._traced_run( + tmp_path / "differ", source_tools=["convert_units"], target_tools=["convert_currency"] + ) + assert "trace-target" not in self._top(render_html(build_report_payload(same))) + assert "trace-target" in self._top(render_html(build_report_payload(differ))) + + def test_no_tool_calls_on_both_sides_is_not_flagged_missing(self, tmp_path: Path) -> None: + neither = self._traced_run(tmp_path / "neither", source_tools=[], target_tools=[]) + dropped = self._traced_run( + tmp_path / "dropped", source_tools=["convert_units"], target_tools=[] + ) + assert "trace-missing" not in self._top(render_html(build_report_payload(neither))) + assert "trace-missing" in self._top(render_html(build_report_payload(dropped))) + + def test_a_within_threshold_finding_is_not_tinted(self, tmp_path: Path) -> None: + html = render_html(build_report_payload(self._threshold_run(tmp_path))) + before, within = self._top(html).split('
', 1) + assert '
' in within + assert "notice-neutral" not in before + def test_report_json_carries_the_counts_and_the_within_threshold_rows( self, tmp_path: Path ) -> None: @@ -3193,3 +3247,137 @@ def test_report_json_carries_the_counts_and_the_within_threshold_rows( assert [r["example_id"] for r in section["within_threshold_regressions"]] == ["ex1"] # The same per-row shape as top_regressions. assert set(section["within_threshold_regressions"][0]) == set(section["top_regressions"][0]) + + +# --------------------------------------------------------------------------- +# Long tables show their first rows and a "show more" toggle +# --------------------------------------------------------------------------- + + +class TestLongTablesShowTheirFirstRows: + """The per-example and trace-rule tables fold rows past ``TABLE_ROWS_SHOWN``.""" + + @staticmethod + def _examples_run(tmp_path: Path, count: int) -> Path: + """A run with ``count`` scored source/target pairs.""" + cwd, run_id = _scaffold_full_run(tmp_path) + run_dir = cwd / ".evalshift" / "runs" / run_id + (run_dir / "raw.jsonl").unlink() + records = [] + for i in range(count): + example_id = f"e{i:03d}" + for role in ("source", "target"): + append_call( + run_dir, + Call( + run_id=run_id, + prompt_id="greet", + example_id=example_id, + model_id=f"model-{role}", + role=role, + text="hi", + cost_usd=0.0001, + latency_ms=100, + input_tokens=10, + output_tokens=5, + ), + ) + records.append( + EvalRecord( + run_id=run_id, + prompt_id="greet", + example_id=example_id, + evaluator_name="structural.length", + source_score=1.0, + target_score=0.9, + delta=-0.1, + ) + ) + (run_dir / SCORES_FILENAME).write_text( + "".join(r.model_dump_json() + "\n" for r in records), encoding="utf-8" + ) + return run_dir + + @staticmethod + def _violations_run(tmp_path: Path, count: int) -> Path: + violation = { + "rule_id": "no-v1", + "rule_type": "forbidden", + "tool": "refund_v1", + "round_index": 0, + "detail": "called forbidden tool 'refund_v1'", + } + return _run_dir_with_scores( + tmp_path, + [ + { + "run_id": "r1", + "prompt_id": "p", + "example_id": f"v{i:03d}", + "evaluator_name": "payments", + "kind": "trace_invariants", + "source_score": 1.0, + "target_score": 0.0, + "delta": -1.0, + "metadata": { + "rules_checked": ["no-v1"], + "source_violations": [], + "target_violations": [violation], + }, + } + for i in range(count) + ], + ) + + @staticmethod + def _block(html: str, heading: str) -> str: + """From ``heading`` to the next block's heading.""" + return html.split(heading, 1)[1].split("

", 1)[0] + + def test_rows_past_the_first_ten_are_folded(self, tmp_path: Path) -> None: + from evalshift_cli.reports.html import TABLE_ROWS_SHOWN + + run_dir = self._examples_run(tmp_path, TABLE_ROWS_SHOWN + 2) + block = self._block( + render_html(build_report_payload(run_dir)), "

Per-example breakdown

" + ) + assert block.count("row-more") == 2 + assert 'type="checkbox"' in block + assert "Show 2 more examples" in block + assert "Show fewer" in block + + def test_one_hidden_row_is_named_in_the_singular(self, tmp_path: Path) -> None: + from evalshift_cli.reports.html import TABLE_ROWS_SHOWN + + run_dir = self._examples_run(tmp_path, TABLE_ROWS_SHOWN + 1) + block = self._block( + render_html(build_report_payload(run_dir)), "

Per-example breakdown

" + ) + assert block.count("row-more") == 1 + assert "Show 1 more example<" in block + + def test_a_short_table_has_no_toggle(self, tmp_path: Path) -> None: + from evalshift_cli.reports.html import TABLE_ROWS_SHOWN + + run_dir = self._examples_run(tmp_path, TABLE_ROWS_SHOWN) + block = self._block( + render_html(build_report_payload(run_dir)), "

Per-example breakdown

" + ) + assert "row-more" not in block + assert 'type="checkbox"' not in block + + def test_the_trace_rules_table_folds_the_same_way(self, tmp_path: Path) -> None: + from evalshift_cli.reports.html import TABLE_ROWS_SHOWN + + run_dir = self._violations_run(tmp_path, TABLE_ROWS_SHOWN + 3) + block = self._block( + render_html(build_report_payload(run_dir)), "

Trace rules broken

" + ) + assert block.count("row-more") == 3 + assert "Show 3 more broken rules" in block + + def test_printing_shows_every_row(self, tmp_path: Path) -> None: + cwd, run_id = _scaffold_full_run(tmp_path) + html = render_html(build_report_payload(cwd / ".evalshift" / "runs" / run_id)) + print_rules = html.split("@media print", 1)[1].split("\n}\n", 1)[0] + assert ".row-more" in print_rules