fix: stop scoring narration beside tool calls; hide untested effect placeholders

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **`semantic` and `llm_judge` no longer score a turn that ended in a tool
+  call.** Without tool-result fixtures the replay stops at the call, so the
+  text beside it is narration ("Let me check!"), not an answer. On a real
+  planner run the source narrated beside four of its calls and the target did
+  not, and both evaluators scored each of those as a −1.0 loss: 8 of the 20
+  regressions behind an `inconclusive` verdict. Such pairs now write no record
+  and count as not applicable; the tool evaluators measure them.
+- **The report's evaluator table no longer prints a skipped test's
+  placeholders as measurements.** A uniform −1.000 drop read
+  "0.00 negligible [0.00, 0.00]"; Effect and 95% range now read "—" when no
+  test ran, and the executive summary drops the "negligible" label there.
+  Effect is shown signed, so −0.44 no longer sits beside a range of
+  [−0.70, −0.22] as an unsigned 0.44.
+
 ## [1.6.1] - 2026-10-23
 
 ### Changed

DOCS.md

@@ -530,11 +530,13 @@ Embeds both outputs and scores the target by cosine similarity to the source (so
 
 Presentation is stripped before embedding, so formatting never reads as drift: a code fence wrapping the **whole** output (` ```json … ``` `) is dropped, and an output that is a JSON object or array is re-serialized canonically (sorted keys, uniform separators). A fenced, pretty-printed object and the same object returned compact score 1.0; a fence inside prose is part of the answer and kept. An empty fenced block counts as an empty output. Whether the output is *valid, unfenced* JSON is a separate question — `json_schema` and the judge answer it.
 
+Both outputs empty (a tool-only turn) writes no record and makes no embedding call. One empty side still scores 0.0 — a target that went silent where the source answered is the regression this evaluator exists for. A turn that **ended in a tool call** on either side is skipped the same way: the replay stops at the call, so any text beside it is narration ("Let me check!"), not an answer, and comparing it scored a target that called tools without narrating as a total loss. The tool evaluators measure those turns.
+
 On an agent turn where **both** models answered with tool calls and no prose, there is nothing to embed — the evaluator writes **no record at all** (no provider call) rather than erroring on an empty embedding input or inventing a score. A turn where only *one* side is empty is still scored: a target that went silent where the source answered is exactly the regression this evaluator exists to catch. Since the empty side cannot be embedded (the provider 400s on empty input), the pair scores **0.0 similarity by definition** with no embedding call — still gated by `min_similarity` as usual — and the record carries `empty_side: "source" | "target"` metadata plus an explanation the report shows verbatim. The mirrored case (source silent, target answered) scores identically.
 
 ### LLM-as-judge (`evaluators.llm_judge`, list)
 
-Pairwise A/B comparison per criterion: the judge sees the two outputs anonymised and **order-randomised** (positional-bias mitigation) and picks a winner or a tie. Winner → scores (0, 1); tie → (0.5, 0.5). Config: `criterion_name`, `criterion_prompt`, `judge_model` (per criterion; built-in default `gemini-3.1-flash-lite-preview`). Write criteria symmetric (never mention "source"/"target") and include an explicit tie instruction; prefer a judge from a third model family so it isn't grading its own relatives. Multi-turn examples render the transcript to the judge (capped at 4,000 chars). Tool-only turns — both outputs empty — write **no record at all** and spend no judge call, since comparing two empty strings only ever returns a meaningless tie, and a fabricated tie is indistinguishable from a judged one.
+Pairwise A/B comparison per criterion: the judge sees the two outputs anonymised and **order-randomised** (positional-bias mitigation) and picks a winner or a tie. Winner → scores (0, 1); tie → (0.5, 0.5). Config: `criterion_name`, `criterion_prompt`, `judge_model` (per criterion; built-in default `gemini-3.1-flash-lite-preview`). Write criteria symmetric (never mention "source"/"target") and include an explicit tie instruction; prefer a judge from a third model family so it isn't grading its own relatives. Multi-turn examples render the transcript to the judge (capped at 4,000 chars). Tool-only turns — both outputs empty — write **no record at all** and spend no judge call, since comparing two empty strings only ever returns a meaningless tie, and a fabricated tie is indistinguishable from a judged one. So is a turn that **ended in a tool call** on either side is skipped the same way: the replay stops at the call, so any text beside it is narration ("Let me check!"), not an answer, and comparing it scored a target that called tools without narrating as a total loss. The tool evaluators measure those turns.
 
 ### Tool-call evaluators — see [Agent evaluation](#agent-evaluation)
 
@@ -718,7 +720,7 @@ Trace-aware debugging: `evalshift diff case <run> <example>` (side-by-side trace
 
 Full contract in [docs/methodology.md](docs/methodology.md). Per `(prompt, evaluator, slice)` comparison over the paired per-example deltas:
 
-1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. A semantic row within `min_similarity` enters the test as a delta of 0 (noted `k of n rows within the evaluator's own threshold (min_similarity) — tested as no change`); `delta_avg_score` stays the raw mean. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets. A tool evaluator is never handed a pair that was offered no tools at all — like an `applies_to` miss, that pair is not attempted: no cell, no coverage entry, so it counts toward neither the "not applicable" note nor "nothing measured".
+1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, a `semantic`/`llm_judge` pair where either side's turn ended in a tool call, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. A semantic row within `min_similarity` enters the test as a delta of 0 (noted `k of n rows within the evaluator's own threshold (min_similarity) — tested as no change`); `delta_avg_score` stays the raw mean. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets. A tool evaluator is never handed a pair that was offered no tools at all — like an `applies_to` miss, that pair is not attempted: no cell, no coverage entry, so it counts toward neither the "not applicable" note nor "nothing measured".
 2. **Normality screen.** Shapiro–Wilk on the deltas at α = 0.05: normal → **paired t-test**, non-normal → **Wilcoxon signed-rank**. n > 5000 skips the screen (CLT, t-test).
 3. **Effect size.** Paired Cohen's d = mean(Δ)/std(Δ), with a 95% CI — analytical for the t-test, percentile bootstrap (2,000 resamples, seeded, deterministic) for Wilcoxon.
 4. **Multiple comparisons.** Benjamini–Hochberg FDR correction at α = 0.05 across *all* testable comparisons in the run (equivalent to `statsmodels fdr_bh`).
@@ -1035,7 +1037,7 @@ Usually correct behaviour: no statistically significant difference. Check n (n <
 
 Neither reading changes the severity, so neither gates the verdict. A zero-variance row with no shift (every pair scored the same) reads **✓ Equivalent — Both models scored the same on every pair.**
 
-The report's **Confidence** column turns the corrected p-value into **Certain** (< 0.01), **Likely** (< 0.05) or **Unclear**. A row where no test ran shows **—**: too few pairs, or every pair scoring the same. Its p-value is a placeholder 1.0, not evidence.
+The report's **Confidence** column turns the corrected p-value into **Certain** (< 0.01), **Likely** (< 0.05) or **Unclear**. A row where no test ran shows **—**: too few pairs, or every pair scoring the same. Its p-value is a placeholder 1.0, not evidence. The **Effect** and **95% range** columns show **—** on that row too: d and its interval are placeholder zeros there, and printed as "0.00 negligible [0.00, 0.00]" they called a uniform −1.000 drop negligible. Effect is shown signed (−0.44, not 0.44), so it sits inside its own range.
 
 ### The policy verdict is `inconclusive`
 

docs/evaluators.md

@@ -108,6 +108,14 @@ sentence ("which output preserves more factual detail?"). Multiple
 Tool-only turns (both outputs empty) are skipped without spending a
 judge call.
 
+Both text evaluators (`semantic.cosine` and `llm_judge`) also skip a
+pair where **either side's turn ended in a tool call**. Without
+tool-result fixtures the replay stops at the call, so text beside it is
+narration ("Let me check!"), not the answer a user would read. Scoring
+it marked a target that called tools without narrating as a total loss.
+The tool evaluators measure those turns; coverage counts the pair as
+not applicable.
+
 Pick the judge from a **third model family**. A judge prefers output
 that reads like its own (self-preference bias), and A/B randomisation
 does nothing against that. `doctor` and `validate` warn when a

docs/methodology.md

@@ -122,7 +122,9 @@ severity, and so the verdict, is unchanged.
 The report's Confidence column reads the adjusted p-value: **Certain** below
 0.01, **Likely** below 0.05, otherwise **Unclear**. A row where no test ran
 (n < 5, or zero variance) shows **—** instead, because its p-value is a
-placeholder 1.0.
+placeholder 1.0. Its Effect and 95% range columns show **—** for the same
+reason: d and its interval are placeholder zeros. Effect is printed signed,
+like its range.
 
 ### Drift inside `min_similarity` is tested as no change
 

llms-full.txt

@@ -809,8 +809,12 @@ evaluators:                         # every evaluator config also takes blocking
     #   no provider call. ONE empty side still scores: 0.0 similarity by definition,
     #   no embedding call (provider 400s on empty input); min_similarity gate applies;
     #   metadata.empty_side = "source"|"target"; either direction scored the same.
+    # either side's turn ENDED IN A TOOL CALL (last round has calls) -> NO record, no
+    #   provider call (semantic AND llm_judge): text beside a call is narration, not an
+    #   answer; the replay stops at the call. Tool evaluators measure those turns.
   llm_judge:                        # list; pairwise A/B, order-randomized; win->(0,1), tie->(.5,.5)
     # both outputs empty (tool-only turn) -> NO record at all, no judge call
+    # either side's turn ended in a tool call -> NO record, no judge call (see semantic)
     - criterion_name: str
       criterion_prompt: str         # keep symmetric + explicit tie instruction
       judge_model: str = "gemini-3.1-flash-lite-preview"  # per-criterion; defaults.judge_model is NOT consulted
@@ -1884,7 +1888,8 @@ All severities "none" -> usually genuinely no significant difference; check n (<
   "uniform drop:"/"uniform gain:", d is a placeholder 0) reads "⚠ Uniform drop/gain".
   Display only: severity, and so the verdict, is unchanged.
   Report "Confidence" = corrected p: Certain (<0.01), Likely (<0.05), Unclear; "—" when no
-  test ran (n<5, or zero variance: p is a placeholder 1.0, not evidence).
+  test ran (n<5, or zero variance: p is a placeholder 1.0, not evidence). Effect and 95% range
+  also read "—" on such a row (d and CI are placeholder 0s); Effect is shown SIGNED, like its range.
 Verdict "inconclusive" -> (1) all evaluators advisory (fresh init: flip blocking: true as the
 suite grows; recommendations name each advisory judge and whether its smallest per-prompt n has
 reached 20, or which prompt is short), (2) rate-budget breach unconfirmed by Wilson CI (grow the suite), (3) all n<5,

src/evalshift_cli/cli/commands/evaluate.py

@@ -121,6 +121,10 @@
 
 SCORES_FILENAME: str = "scores.jsonl"
 
+#: Text evaluators that compare the two models' *answers*. A turn that ended
+#: in a tool call has none yet, so these skip it; structural checks still run.
+_ANSWER_KINDS: frozenset[str] = frozenset({"semantic", "llm_judge"})
+
 
 @dataclass(frozen=True, slots=True)
 class _PairedCalls:
@@ -1346,6 +1350,15 @@ async def _score_one(
     if _is_tool_evaluator(evaluator):
         return await _score_one_tool(evaluator, pair, run_id, examples_by_id)
 
+    # A turn that ended in a tool call has no answer yet: the replay stops
+    # at the call, and any text beside it is narration. Comparing narration
+    # (or its absence) scored a target that called tools without chatting
+    # as a total loss. The tool evaluators above measure these turns.
+    if _evaluator_kind(evaluator) in _ANSWER_KINDS and any(
+        c.trace is not None and c.trace.ends_in_tool_calls for c in (pair.source, pair.target)
+    ):
+        return []
+
     example = examples_by_id.get(pair.example_id)
     input_vars = example.inputs if example is not None else {}
     history = (

src/evalshift_cli/evaluators/tool_models.py

@@ -205,6 +205,16 @@ def call_count(self) -> int:
         """Number of tool calls in this trace."""
         return len(self.calls)
 
+    @property
+    def ends_in_tool_calls(self) -> bool:
+        """True iff the last response called a tool, so it awaits results.
+
+        Text beside such a call is narration ("Let me check!"), never the
+        answer a user would read: that answer comes after the results.
+        """
+        last = self.round_count - 1
+        return any(c.round_index == last for c in self.calls)
+
     @property
     def tool_names(self) -> list[str]:
         """Tool names in the order they were called."""

src/evalshift_cli/reports/templates/report.html.j2

@@ -531,9 +531,9 @@
                 <span class="verdict-blurb">{{ blurb }}</span>
               </span>
             </td>
-            <td class="num" title="Δ avg score {{ "%+.3f"|format(row.delta_avg_score) }} · Cohen's |d| {{ "%.2f"|format(row.effect_size|abs) }}">
+            <td class="num" title="Δ avg score {{ "%+.3f"|format(row.delta_avg_score) }}{% if row.test != "skipped" %} · Cohen's d {{ "%+.2f"|format(row.effect_size) }}{% endif %}">
               {{ "%+.0f"|format(row.delta_avg_score * 100) }}%
-              <span class="magnitude" style="display:block">{{ effect_magnitude(row.effect_size) }}</span>
+              {% if row.test != "skipped" %}<span class="magnitude" style="display:block">{{ effect_magnitude(row.effect_size) }}</span>{% endif %}
             </td>
             {% set conf = confidence_label(row.severity, row.p_value_corrected, row.test) %}
             <td class="num" title="{% if row.test == "skipped" %}No test ran: too few pairs, or every pair scored the same{% else %}p_corrected = {{ "%.3f"|format(row.p_value_corrected) }} (Benjamini–Hochberg){% endif %}">
@@ -773,8 +773,14 @@
               <td class="num faint">—</td>
               {% else %}
               <td class="num {{ "bad" if row.delta_avg_score < 0 else "ok" if row.delta_avg_score > 0 else "" }}">{{ "%+.3f"|format(row.delta_avg_score) }}</td>
-              <td class="num">{{ "%.2f"|format(row.effect_size|abs) }} <span class="magnitude">{{ effect_magnitude(row.effect_size) }}</span></td>
+              {% if row.test == "skipped" %}
+              {# No test ran: d and its interval are placeholder zeros, not measurements. #}
+              <td class="num faint">—</td>
+              <td class="num faint">—</td>
+              {% else %}
+              <td class="num">{{ "%.2f"|format(row.effect_size) }} <span class="magnitude">{{ effect_magnitude(row.effect_size) }}</span></td>
               <td class="num muted ci">[{{ "%.2f"|format(row.effect_size_ci_low) }}, {{ "%.2f"|format(row.effect_size_ci_high) }}]</td>
+              {% endif %}
               <td class="num">{% if conf == "—" %}<span class="faint">—</span>{% else %}<span class="badge conf conf-{{ conf|lower }}">{{ conf }}</span>{% endif %}</td>
               {% endif %}
               <td>
@@ -823,7 +829,7 @@
               </td>
               <td class="num">{{ row.n }}</td>
               <td class="num {{ "bad" if row.delta_avg_score < 0 else "ok" if row.delta_avg_score > 0 else "" }}">{{ "%+.3f"|format(row.delta_avg_score) }}</td>
-              <td class="num">{{ "%.2f"|format(row.effect_size|abs) }} <span class="magnitude">{{ effect_magnitude(row.effect_size) }}</span></td>
+              <td class="num">{{ "%.2f"|format(row.effect_size) }} <span class="magnitude">{{ effect_magnitude(row.effect_size) }}</span></td>
               <td class="num">{% if conf == "—" %}<span class="faint">—</span>{% else %}<span class="badge conf conf-{{ conf|lower }}">{{ conf }}</span>{% endif %}</td>
               <td><span class="verdict-head">{{ verdict_glyph(row.severity, row.notes, row.effect_size) }} {{ verdict(row.severity, row.notes, row.effect_size)[0] }}</span></td>
             </tr>