feat(cache): scope the response cache to the project, add --no-cache

CHANGELOG.md

@@ -42,6 +42,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   didn't") with no tool call in its turn — this case replays with that claim
   in context.`). The case is still promoted. A recorded history with no tool
   rows at all, and history reconstructed from sibling turns, are not checked.
+- **`run --no-cache` and `compare --no-cache`** send every run-stage call live
+  instead of replaying cached responses. The fresh responses still go into the
+  cache and replace the ones they bypassed, so the next ordinary run replays
+  the new answers, not the old ones. The evaluate stage's judge and embedding
+  cache is unaffected.
+- **A run says when it replayed calls from the cache.** A replayed call never
+  reached the model, so a model that changed behind the same name (another
+  server, a server-side system prompt, re-pulled weights) could not show up in
+  it, and nothing said so. `run` now prints `⚠ N model calls were replayed from
+  this project's response cache (T on the target) …` under its summary, and
+  `compare` prints it above the verdict, naming `--no-cache` as the way out.
+
+### Changed
+
+- **The response cache belongs to the project.** It moved from the
+  machine-wide `~/.evalshift/cache.db` to `.evalshift/cache.db` in the folder
+  the command runs in, beside `.evalshift/runs/`. A second project testing a
+  different backend under the same model name used to replay the first
+  project's answers. The first run in each project starts with an empty
+  cache. Nothing reads the old file any more; `evalshift cache clear` clears
+  this project's cache, names its path, and points at the old file when it
+  exists (it never deletes it). The default `.gitignore` advice
+  (`.evalshift/*`) already covers the new file.
 
 ## [1.6.2] - 2026-10-27
 

DOCS.md

@@ -252,7 +252,7 @@ Run ids look like `r_20260722_golden_a1b2c3` (`r_<date>_<suite-slug>_<hex>`). Un
 
 ### Response cache
 
-Live run-stage responses are cached in SQLite at `~/.evalshift/cache.db`, keyed by SHA-256 over canonical JSON of `(model, prompt, inputs, temperature, max_tokens[, history])` — plus `generation_config`, the tools array as sent, the round index and the sample index when each is set — with a 7-day TTL. Re-running an identical evaluation costs no run-stage calls, agent suites included. An example with a non-empty toolset gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, **the exact message list that round sends** (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and `strict` — reordering the tools is a miss), `generation_config` (so `tool_choice` and `parallel_tool_calls`), the effective temperature and `max_tokens`, the round index and the sample index — editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (calls with their ids and arguments, final text, refusal), tokens, cost, latency and finish reason, so the `raw.jsonl` row matches the live one apart from `cached` and `cached_rounds`; a multi-round example counts as cached only when every round was a hit. A row with any round served from the cache (`cached_rounds > 0`) carries latency recorded on an earlier run, so it is left out of the report's live latency figures and its latency delta is marked not comparable. Errors are never cached (the next run re-sends a failed round; the rounds before it come from the cache), truncated responses are cached and stay flagged, and a hit records the original call's cost and latency. A `cache.db` written by an earlier version is upgraded in place on open and keeps its entries; several `evalshift` processes may open it at once. Disable per-project with `defaults.cache: false`; wipe with `evalshift cache clear`.
+Live run-stage responses are cached in SQLite at `.evalshift/cache.db` in the folder the command runs in (beside `.evalshift/runs/`), so two projects never replay each other's answers. Entries are keyed by SHA-256 over canonical JSON of `(model, prompt, inputs, temperature, max_tokens[, history])` — plus `generation_config`, the tools array as sent, the round index and the sample index when each is set — with a 7-day TTL. Re-running an identical evaluation costs no run-stage calls, agent suites included. An example with a non-empty toolset gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, **the exact message list that round sends** (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and `strict` — reordering the tools is a miss), `generation_config` (so `tool_choice` and `parallel_tool_calls`), the effective temperature and `max_tokens`, the round index and the sample index — editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (calls with their ids and arguments, final text, refusal), tokens, cost, latency and finish reason, so the `raw.jsonl` row matches the live one apart from `cached` and `cached_rounds`; a multi-round example counts as cached only when every round was a hit. A row with any round served from the cache (`cached_rounds > 0`) carries latency recorded on an earlier run, so it is left out of the report's live latency figures and its latency delta is marked not comparable. Errors are never cached (the next run re-sends a failed round; the rounds before it come from the cache), truncated responses are cached and stay flagged, and a hit records the original call's cost and latency. A `cache.db` written by an earlier version is upgraded in place on open and keeps its entries; several `evalshift` processes may open it at once. A replayed call never reached the model, so it cannot show a model that changed behind the same name (another server behind the same id, a system prompt injected server-side, re-pulled weights): when any calls were replayed, `run` prints `⚠ N model calls were replayed from this project's response cache (T on the target) …` under its summary, and `compare` prints the same line above the verdict. `run --no-cache` / `compare --no-cache` send every run-stage call live and still write the fresh responses, replacing the cached ones, so the next ordinary run replays the new answers rather than the bypassed ones; the evaluate stage's judge and embedding cache is unaffected. Disable per-project with `defaults.cache: false` (no reads, no writes); wipe with `evalshift cache clear`. EvalShift 1.6.2 and earlier kept one machine-wide cache at `~/.evalshift/cache.db`; nothing reads it now, `cache clear` points at it when it exists, and it can be deleted.
 
 The cache covers the evaluate stage too: `semantic` embeddings are keyed by `(embedding model, text)`, and `llm_judge` verdicts by `(judge model, criterion, source output, target output)`. The judge key uses a canonical A/B ordering, so the per-call orientation randomization doesn't halve the hit rate — the orientation that was actually used is recorded with the verdict and replayed on a hit, leaving `metadata.target_was_a` faithful.
 
@@ -912,7 +912,8 @@ Without `--ci`, warns after writing when an existing workflow under `.github/wor
 **`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config or a blocking `semantic` / `llm_judge` model with no API key. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`warn` naming the package to install when the client library is missing, `warn` when the bucket cannot be listed), and an `evaluator keys` row when the config has a `semantic` or `llm_judge` evaluator (`fail` for a blocking one whose model has no key, `warn` for an advisory one, `ok` otherwise; top-level block and every named suite). Adds a `captures limits` row (`warn`) when `captures.store` is set without `captures.max_cases`, and, with either capture limit set, a `suite freshness` row per `source: captured` suite (`ok`, or `warn` per finding; never a failure).
 
 **`evalshift run`** — paired evaluation run (costs money — calls real models).
-`-f/--from <model>` · `-t/--to <model>` · `-c/--config` · `-s/--suite <file>` · `--suite-name <name>` · `--resume` · `-y/--yes`
+`-f/--from <model>` · `-t/--to <model>` · `-c/--config` · `-s/--suite <file>` · `--suite-name <name>` · `--resume` · `-y/--yes` · `--no-cache` (send every call live; fresh responses replace the cached ones — see [Response cache](#response-cache))
+Prints a `⚠` line under the summary when any calls were replayed from the cache.
 
 **`evalshift evaluate <run-id>`** — score all pairs → `scores.jsonl`. `-c/--config`
 Prints a red doctor-style **broken eval harness** row when the *source* model failed the recorded ground truth on at least half of at least four `tool_selection.conformance` rows — the suite's expectations were captured from the source model, so the source is the one side that should satisfy them, and when it does not the run measured the harness rather than the migration. It names the rate (`10 of 10 … (100%)`) and the likely causes. `evalshift compare` prints the same row immediately above the verdict block. Below four conformance rows the check stays silent: a 100% rate over three has a 95% Wilson lower bound of 0.44 and cannot support the claim.
@@ -925,7 +926,7 @@ On an interactive terminal, `report` and `compare` end with one line asking for
 
 **`evalshift compare`** — full pipeline under one live display: two models, **one** suite. It takes a single `--suite-name`/`--suite`, auto-selects when exactly one suite is wired, and prints a ready-to-run command per suite when several are. It does not run every suite — loop over the names, or use one CI job per suite. Formerly `all`; that name is hidden but still works, since scaffolded `EVALSHIFT.md` files reference it.
 All `run` flags, plus `--gate` · `--policy-gate` · `--open` · `--push` · `--insights/--no-insights`
-With a capture limit set, prints the `suite freshness` findings for a named or auto-selected `source: captured` suite as `⚠` lines above the verdict.
+With a capture limit set, prints the `suite freshness` findings for a named or auto-selected `source: captured` suite as `⚠` lines above the verdict. When any run-stage calls were replayed from the cache, prints the replay `⚠` line above the verdict too.
 
 ### Hosted
 
@@ -955,7 +956,7 @@ With `captures.max_age` or `captures.max_cases` set, promotes only captures reco
 ### Housekeeping
 
 **`evalshift runs clean`** — `--keep <n>` · `--older-than <days>` · `--suite <slug>` · `--dry-run` · `-y/--yes` · `--config`
-**`evalshift cache clear`** — wipe the response cache.
+**`evalshift cache clear`** — wipe this project's response cache (`.evalshift/cache.db` in the current folder) and name the path it cleared. When the old machine-wide `~/.evalshift/cache.db` exists it says so; it never deletes it.
 
 ### Hidden debug commands
 

docs/configuration.md

@@ -417,7 +417,7 @@ A list of prompt definitions. Each entry has:
 | `judge_model`   | string | `gemini-3.1-flash-lite-preview` | Default LLM-as-judge model. |
 | `insights_model`| string | (none)                        | Model that writes the run-insights narrative rendered in `report.html` and uploaded with the bundle. Falls back to `judge_model` when unset — writing analytical prose is a harder task than a pairwise A/B verdict, so it is worth tuning separately. See [Run insights](#run-insights). |
 | `concurrency`   | int    | 10 (1 ≤ x ≤ 64)               | Max in-flight LLM calls during `evalshift run` **and** `evalshift evaluate` (the embedding and judge calls made while scoring). |
-| `cache`         | bool   | `true`                        | Read/write the local SQLite cache at `~/.evalshift/cache.db`. Covers run-stage completions — for examples that offer tools, one entry per replayed round — plus `semantic` embeddings and `llm_judge` verdicts. See [Response cache](https://github.com/evalshift/evalshift-cli/blob/main/DOCS.md#response-cache) for the key. |
+| `cache`         | bool   | `true`                        | Read/write the project's SQLite cache at `.evalshift/cache.db` (in the folder the command runs in). `run`/`compare --no-cache` bypass reads for one run while still writing. Covers run-stage completions — for examples that offer tools, one entry per replayed round — plus `semantic` embeddings and `llm_judge` verdicts. See [Response cache](https://github.com/evalshift/evalshift-cli/blob/main/DOCS.md#response-cache) for the key. |
 | `max_cost_usd`  | float  | 50.0                          | Soft ceiling reserved for future enforcement. The pre-flight cost prompt currently triggers above $10 (skip with `--yes`). |
 | `max_tokens`    | int    | 4096 (`> 0`)                  | Completion length cap sent to every model call. Raise it if outputs are being truncated (the provider returns `finish_reason == "length"`); a `prompts[].max_tokens` entry overrides it per prompt. Truncated calls are detected, surfaced in the report, and **excluded from the regression statistics** so a cut-off output can't manufacture a false regression. |
 | `samples_per_example` | int | 1 (1 ≤ x ≤ 20)           | How many times each `(prompt, example)` is sent to **each** model. Above 1, every sample is its own live call (the cache keys on the sample index), sample *i* of the source is scored against sample *i* of the target, and the example's row in `scores.jsonl` becomes the **mean over samples** with the per-sample scores and the within-example `delta_variance` under `metadata.samples`. The paired tests still run over examples, not samples, so this reduces noise without inflating `n`. Cost and the call count multiply by it; only worth turning on for a model that samples non-deterministically (see the report banner). See [Methodology](methodology.md#limitations-to-be-aware-of). |

docs/faq.md

@@ -7,7 +7,7 @@
 machine to the LLM provider you configured — any provider LiteLLM supports —
 using your own API keys.
 
-The local SQLite cache at `~/.evalshift/cache.db` only contains
+The project's SQLite cache at `.evalshift/cache.db` only contains
 provider responses for *your* prompts and inputs.
 
 Hosted uploads are explicit. `bundle` packages the completed

docs/getting-started.md

@@ -38,8 +38,9 @@ fastest.
 
 EvalShift calls the provider you configure — any provider LiteLLM supports — directly using your own keys.
 Local runs do not send prompts or outputs to an EvalShift-operated server.
-Provider responses are cached locally in `~/.evalshift/cache.db`, so re-running an
-unchanged suite (agent suites included) makes no new calls.
+Provider responses are cached in the project at `.evalshift/cache.db`, so re-running an
+unchanged suite (agent suites included) makes no new calls. The run says how many calls it
+replayed; pass `--no-cache` to send them all live.
 
 Set whichever providers you intend to use:
 

docs/index.md

@@ -90,7 +90,7 @@ Housekeeping:
 ```
 evalshift inspect <run-id> [--failed]   # per-row deltas (--failed: negative or errored only)
 evalshift bundle <run-id> -o out.json.gz   # write the bundle somewhere else
-evalshift cache clear                   # wipe the local response cache
+evalshift cache clear                   # wipe this project's response cache
 evalshift runs clean                    # prune old runs (--keep, --older-than, --dry-run)
 ```
 

llms-full.txt

@@ -101,7 +101,8 @@ as one section under the pipeline block (also on stage failure); errors are neve
 | traces.jsonl | traces import | optional BYO agent traces |
 | run_bundle.json.gz | bundle/push | optional hosted upload bundle |
 
-Cache: SQLite ~/.evalshift/cache.db, key = SHA-256 of canonical JSON {model, prompt, inputs,
+Cache: SQLite .evalshift/cache.db in the folder the command runs in (beside .evalshift/runs/;
+projects never share it), key = SHA-256 of canonical JSON {model, prompt, inputs,
 temperature, max_tokens[, history]} + generation_config, tools array as sent, round index, sample
 index when set; TTL 7 days. Tool-calling examples ARE cached, one entry per replayed round, keyed
 on canonical model + prompt + inputs + the exact message list that round sends (history, current
@@ -115,7 +116,15 @@ cached_rounds > 0 = latency partly from an earlier run: excluded from live laten
 latency_comparable false. Errors never cached (next run re-sends the failed round; earlier rounds
 hit); truncated responses cached and still flagged; a hit records the original cost and latency.
 An older cache.db is upgraded in place and keeps its entries; concurrent opens by several
-processes are safe. defaults.cache: false disables; `evalshift cache clear` wipes.
+processes are safe. A replayed call never reached the model, so it cannot show a model changed
+behind the same name (other server, server-side system prompt, re-pulled weights): when any
+call was replayed, `run` prints "⚠ N model calls were replayed from this project's response cache
+(T on the target) ..." under its summary and `compare` prints it above the verdict.
+`run/compare --no-cache` sends every run-stage call live AND writes the fresh responses (replacing
+the cached ones), so the next ordinary run replays the new answers; evaluate-stage judge/embedding
+cache unaffected. defaults.cache: false disables (no reads, no writes); `evalshift cache clear`
+wipes this project's cache. 1.6.2 and earlier used one machine-wide ~/.evalshift/cache.db; nothing
+reads it now, `cache clear` mentions it when present (never deletes it).
 Resume: `run --resume` continues newest in_progress run; requires config_hash match (canonical
 config + suite PATH; suite contents are NOT hashed, so start fresh after editing examples),
 skips (prompt_id, example_id, role, sample_index) keys already in raw.jsonl; errored calls
@@ -225,7 +234,7 @@ evalshift doctor
     "alpha: N case file(s) could not be read; the counts leave them out"
   Measured against defaults.source_model, models matched on bare names (see capture sync).
 evalshift run [-f/--from MODEL] [-t/--to MODEL] [-c CONFIG] [-s/--suite FILE]
-    [--suite-name NAME] [--resume] [-y/--yes]
+    [--suite-name NAME] [--resume] [-y/--yes] [--no-cache]
   Paired run over (prompt x example x {source,target}). --from/--to override
   defaults.source_model/target_model. --suite default ./golden.jsonl; --suite-name selects a
   key under suites: in config. Costs money — always calls real models.
@@ -701,7 +710,7 @@ evalshift diff case RUN_ID EXAMPLE_ID          (trace diff when traces exist, el
 evalshift replay case RUN_ID EXAMPLE_ID [--model source|target (default target)] [--trace]
 evalshift runs clean [--keep N] [--older-than DAYS] [--suite SLUG] [--dry-run] [-y] [--config]
   Precedence for keep count: --keep > EVALSHIFT_MAX_RUNS > config (default 20). 0 disables.
-evalshift cache clear
+evalshift cache clear                          (this project's .evalshift/cache.db)
 
 Hidden debug commands:
 evalshift validate [-s SUITE=golden.jsonl] [-c CONFIG]   (config+suite+prompt cross-check;