babaliauskas · Nov 1, 2026 · Oct 27, 2026 · Oct 27, 2026 · Oct 27, 2026 · Oct 27, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Promotion refuses a turn that claims a tool call it never made.** A turn
+  that was offered tools, called none, and still replied "my notes search
+  didn't find anything" was promoted as ground truth: the fabricated reply
+  became `expected`, and the case asserted `expected_no_tools: true`.
+  `capture promote` now exits 1 on it (`Re-run with --allow-unbacked-claims to
+  promote it anyway.`), and `capture sync` skips it and counts it on its own
+  (`skipped 1 capture(s) whose reply claims a tool call it never made`). The
+  check reads the reply's English wording — a first-person claim that the
+  action already happened, such as "I searched", "I've checked" or "my notes
+  search didn't"; a promise ("let me check", "I'll search your notes once you
+  confirm") or a bare tool name never fires — and never flags a turn with any
+  other sign it acted: a tool call or tool result, a requested tool call, a
+  retrieval event, or more than one model call.
+  `capture promote --allow-unbacked-claims`, `capture sync
+  --allow-unbacked-claims/--block-unbacked-claims` and
+  `captures.allow_unbacked_claims` in `evalshift.yaml` (flag, then YAML, then
+  block) promote it with a warning instead; such a case never asserts
+  `expected_no_tools`. `--allow-errored` does not apply, and an errored turn
+  is still reported as errored. Cases an earlier sync promoted stay in the
+  suite: sync never deletes a case file, so its skip line names the file to
+  delete (`Its case is still in the suite from an earlier sync: delete
+  .evalshift/suites/<suite>/<capture_id>.json to drop it.`), unless the case
+  is marked `provenance: reviewed`. `capture sync --repair` refusing such a
+  case adds `Re-run with --allow-unbacked-claims to repair it anyway.` A refused turn
+  also drops out of later turns' reconstructed history. `init` scaffolds the
+  key commented out. Older CLIs reject the key, so update a CI
+  `evalshift-version:` pin first.
+- Promotion warns when a case's recorded history carries an assistant reply
+  that claims a tool action with no tool call or tool result anywhere in its
+  turn (`history message 2 (assistant) claims a tool action ("my notes search
+  didn't") with no tool call in its turn — this case replays with that claim
+  in context.`). The case is still promoted. A recorded history with no tool
+  rows at all, and history reconstructed from sibling turns, are not checked.
+
 ## [1.6.2] - 2026-10-27
 
 ### Changed

diff --git a/DOCS.md b/DOCS.md
diff --git a/README.md b/README.md
@@ -7,6 +7,10 @@ Open-source LLM migration and regression testing for AI agents.
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
 [![PyPI](https://img.shields.io/pypi/v/evalshift.svg)](https://pypi.org/project/evalshift/)
 
+[![EvalShift getting started: catch model migration regressions in 4 minutes](https://img.youtube.com/vi/22-dRnu5fXs/sddefault.jpg)](https://www.youtube.com/watch?v=22-dRnu5fXs)
+
+▶ [Watch the 4-minute getting-started video on YouTube](https://www.youtube.com/watch?v=22-dRnu5fXs)
+
 **The SDK captures what your agent really did. The CLI replays it against a
 candidate model and tells you what broke. The hosted app keeps the history.**
 

diff --git a/docs/agents.md b/docs/agents.md
@@ -404,7 +404,7 @@ truth.
 ## What does not belong in a golden suite
 
 Ground truth is what the agent *did right*, and a capture is not automatically
-that. Three shapes are actively filtered or flagged during promotion:
+that. These shapes are actively filtered or flagged during promotion:
 
 - **Errored turns are refused.** A capture whose trace carries an `error` event
   died before the agent finished — an upstream 400, a timeout, a crash. Promoted
@@ -413,6 +413,36 @@ that. Three shapes are actively filtered or flagged during promotion:
   `capture sync` skips it, both naming the first error. `--allow-errored`
   promotes it anyway; even then the case never gets `expected_no_tools: true`,
   because a turn that never ran is not evidence that inaction was right.
+- **Turns that claim a tool call they never made are refused.** A turn that was
+  offered tools, called none, and still replied "I don't have that
+  information — my notes search didn't find anything" records an action that
+  never happened. Promoted, the fabricated reply becomes `expected` and the
+  turn asserts `expected_no_tools: true`. The check reads the reply's wording:
+  an English first-person claim that the action already happened, such as "I
+  searched", "I've checked" or "my notes search didn't". A promise ("let me
+  check", "I'll search your notes once you confirm") never fires — a
+  clarifying question is a correct no-tool turn — nor does a bare tool name.
+  A turn with any other sign it acted is never flagged: a recorded tool call or
+  tool result, a requested tool call, a `retrieval` event, or more than one
+  model call (an agent loop):
+
+  ```text
+  ⚠ skipped cap_bad: turn was offered tools and called none, but its reply claims a tool action ("my notes search didn't") — promoting it would make a hallucinated action ground truth.
+  ✓ promoted 1 capture(s) into 1 suite(s), skipped 1 capture(s) whose reply claims a tool call it never made (use --allow-unbacked-claims or captures.allow_unbacked_claims to promote anyway).
+  ```
+
+  `capture promote` exits non-zero with `Re-run with --allow-unbacked-claims to
+  promote it anyway.` For `capture sync`, `--allow-unbacked-claims` or
+  [`captures.allow_unbacked_claims: true`](configuration.md#captures) promotes
+  such turns (`--block-unbacked-claims` overrides the YAML for one run); the
+  refusal prints as a warning and the case never gets `expected_no_tools: true`.
+  `--allow-errored` does not help here, and an errored turn is reported as
+  errored. A case an earlier sync promoted stays in the suite — sync never
+  deletes a case file — and, unless it is marked `provenance: reviewed`, the
+  skip line appends `Its case is still in the suite from an earlier sync:
+  delete .evalshift/suites/<suite>/<capture_id>.json to drop it.` Under
+  `--repair`, a case whose rebuild is refused this way prints `can't repair
+  <case>: … Re-run with --allow-unbacked-claims to repair it anyway.`
 - **Captures missing a recorded toolset are refused, unconditionally.** Every
   model call is required to record the toolset it was offered; a capture whose
   first `model_call` has no `toolset_ref` — the SDK failed to write the sidecar,

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -1096,6 +1096,7 @@ captures:
   max_age: 14d       # optional: see Bounded suites below
   max_cases: 200     # optional, per suite
   rounds: all        # optional: replay every recorded round, not just the first call
+  allow_unbacked_claims: true  # optional: promote turns that claim a tool call they never made
 ```
 
 | Field       | Type                              | Default | Description |
@@ -1104,6 +1105,7 @@ captures:
 | `max_age`   | duration (`30m`, `24h`, `14d`; > 0, ≤ `36500d`) | (none)  | Captures recorded longer ago than this never enter a suite through `capture sync` (`capture promote` ignores the limits), and promoted cases that old are stale: with `max_cases` also set, `capture sync` replaces them as fresh captures arrive. Also the window `capture sync` and `capture list` fetch from `store`. Omit for no age limit. |
 | `max_cases` | int (≥ 1)                         | (none)  | The most cases `capture sync` keeps in each captured suite. Also caps what `capture sync` and `capture list` download from `store`: at most `3 × max_cases` captures per suite per run, a conversation at a time. Omit and suites grow with every sync. |
 | `rounds`    | `first` \| `all`                  | `first` | How many recorded agent rounds a case `capture sync` promotes is replayed over. `first` replays the first model call single-shot: a turn that answers after a tool call is scored on that call alone, so `semantic` and `llm_judge` see no answer text. `all` replays every recorded round [teacher-forced](agents.md#agent-rounds-and-what-a-replay-can-reproduce), feeding back the recorded tool results, through the answer round; it costs about one call per round. `capture sync --rounds` overrides it for one run; `capture promote` takes only the flag. |
+| `allow_unbacked_claims` | bool                         | `false` | Whether `capture sync` promotes a turn that was offered tools, called none, and still replied as if it had (`my notes search didn't find anything`). Off, it skips such a turn: promoting it would make a hallucinated action ground truth, with `expected_no_tools: true`. On, the turn is promoted with a warning and never asserts `expected_no_tools`. `capture sync --allow-unbacked-claims` / `--block-unbacked-claims` override it for one run; `capture promote` takes only `--allow-unbacked-claims`. `--allow-errored` does not apply. |
 
 `rounds` applies to the cases `capture sync` promotes from then on: sync never rewrites a promoted
 case unasked. When `rounds` is `all` and promoted cases still replay one round although their
@@ -1116,6 +1118,17 @@ and ones promoted under another name (`capture promote --as`) are left out of th
 evalshift older than 1.6.0 rejects the key, so update a CI pin (`evalshift-version:`) before
 adding it.
 
+`allow_unbacked_claims` judges the reply's wording only: an English first-person claim that the
+action already happened (`I searched`, `I've checked`, `my notes search didn't`), never a
+promise (`let me check`, `I'll search your notes once you confirm` — a clarifying question is a
+correct no-tool turn) and never a bare mention of a tool. A turn with any other sign it acted is
+never flagged: a recorded tool call or tool result, a requested tool call, a `retrieval` event,
+or more than one model call (an agent loop). An errored turn is reported as errored. Sync never
+deletes a case file, so a case an earlier sync promoted from such a capture stays in the suite;
+unless that case is marked `provenance: reviewed`, the skip line names the file to delete (`Its
+case is still in the suite from an earlier sync: delete
+.evalshift/suites/<suite>/<capture_id>.json to drop it.`). A CLI released before the key rejects it; update a CI pin first.
+
 Setting either `max_age` or `max_cases` turns [bounded suites](#bounded-suites) on. Both work
 without `store`: they govern how captures become suites, whether the captures came from disk or
 from a bucket. A `max_age` that is not a duration fails to load; `evalshift validate` reports it as
@@ -1456,8 +1469,9 @@ paired statistics (`--keep-duplicates` opts out). The dedup set is seeded
 from the cases already in the suite dir, so re-syncing after recording more
 captures can't slip a duplicate past it. Captures whose messages carry an
 image or other non-text content are skipped too (see
-[Provider message shapes](conversations.md#provider-message-shapes)), as is
-a capture no valid case can be built from, and
+[Provider message shapes](conversations.md#provider-message-shapes)), as are
+a capture no valid case can be built from and one whose reply claims a tool
+call it never made (see [`captures.allow_unbacked_claims`](#captures)), and
 `--repair` rebuilds cases an earlier sync built from stringified message
 blocks. Useful flags: `--input-var`
 (default `input`), `--suite <name>` to filter to one suite, `--tag`,

diff --git a/docs/conversations.md b/docs/conversations.md
@@ -96,6 +96,31 @@ exchanges are absent), and the system prompt is only recovered when at least
 one turn in the conversation recorded a full messages list. Prefer the
 messages-list convention above when you can.
 
+A turn that is not promoted — refused for any reason: an `error` event, no
+recorded toolset, content a text replay can't carry, a reply claiming a tool
+call it never made — also drops out of later turns' reconstructed history, so
+a fabricated "my notes search didn't find anything" is never stitched into a
+sibling's prefix. A turn promoted anyway (`--allow-errored`,
+`--allow-unbacked-claims`) does seed it.
+
+Recovered history is checked for such claims too. When a turn's own messages
+list carries an assistant reply that claims a tool action with no tool call
+or tool result anywhere in its turn — from one user message to the next, so a
+reply that says "I searched your calendar." just before the call is backed —
+promotion warns and still promotes. The current turn may be fine, but the
+candidate will answer with that claim in context:
+
+```text
+⚠ history message 2 (assistant) claims a tool action ("my notes search didn't") with no tool call in its turn — this case replays with that claim in context.
+```
+
+A `user` message directly after a `tool` message does not start a new turn:
+Claude delivers tool results inside a user turn. A recorded history with no
+tool rows at all is not checked — the app may have stripped them when it
+resent the conversation, and then every earlier claim would look unbacked.
+Reconstructed history is never checked for the same reason: it carries no
+tool rows.
+
 With `captures.max_cases` set, a conversation's new turns enter a suite
 together when they fit; one larger than the suite's free slots is skipped
 for the next one. Once no whole conversation fits, the slots left take the
@@ -181,7 +206,9 @@ they are promoted and even past `--since`. A case whose capture is gone from
 both `.evalshift/captures/` and the store is named (`can't repair N case(s)`):
 delete it or re-capture. A case with stringified blocks whose rebuild is now
 refused — its turn carries an image, say — is named too
-(`can't repair <case>: …`) and left as it is; a case left with empty `inputs`
+(`can't repair <case>: …`) and left as it is — when the refusal is a reply
+claiming a tool call it never made, the line adds `Re-run with
+--allow-unbacked-claims to repair it anyway.`; a case left with empty `inputs`
 whose rebuild is refused is left as it is too, but reported like any other
 refused capture (`skipped <capture>: …`, counted in the summary). A turn
 whose capture recorded no messages list, or one with no user turn to split