diff --git a/CHANGELOG.md b/CHANGELOG.md index 81d64085..ecbca789 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 didn't") with no tool call in its turn — this case replays with that claim in context.`). The case is still promoted. A recorded history with no tool rows at all, and history reconstructed from sibling turns, are not checked. +- **`run --no-cache` and `compare --no-cache`** send every run-stage call live + instead of replaying cached responses. The fresh responses still go into the + cache and replace the ones they bypassed, so the next ordinary run replays + the new answers, not the old ones. The evaluate stage's judge and embedding + cache is unaffected. +- **A run says when it replayed calls from the cache.** A replayed call never + reached the model, so a model that changed behind the same name (another + server, a server-side system prompt, re-pulled weights) could not show up in + it, and nothing said so. `run` now prints `⚠ N model calls were replayed from + this project's response cache (T on the target) …` under its summary, and + `compare` prints it above the verdict, naming `--no-cache` as the way out. + +### Changed + +- **The response cache belongs to the project.** It moved from the + machine-wide `~/.evalshift/cache.db` to `.evalshift/cache.db` in the folder + the command runs in, beside `.evalshift/runs/`. A second project testing a + different backend under the same model name used to replay the first + project's answers. The first run in each project starts with an empty + cache. Nothing reads the old file any more; `evalshift cache clear` clears + this project's cache, names its path, and points at the old file when it + exists (it never deletes it). The default `.gitignore` advice + (`.evalshift/*`) already covers the new file. ## [1.6.2] - 2026-10-27 diff --git a/DOCS.md b/DOCS.md index 637fed2c..cb54d798 100644 --- a/DOCS.md +++ b/DOCS.md @@ -252,7 +252,7 @@ Run ids look like `r_20260722_golden_a1b2c3` (`r___`). Un ### Response cache -Live run-stage responses are cached in SQLite at `~/.evalshift/cache.db`, keyed by SHA-256 over canonical JSON of `(model, prompt, inputs, temperature, max_tokens[, history])` — plus `generation_config`, the tools array as sent, the round index and the sample index when each is set — with a 7-day TTL. Re-running an identical evaluation costs no run-stage calls, agent suites included. An example with a non-empty toolset gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, **the exact message list that round sends** (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and `strict` — reordering the tools is a miss), `generation_config` (so `tool_choice` and `parallel_tool_calls`), the effective temperature and `max_tokens`, the round index and the sample index — editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (calls with their ids and arguments, final text, refusal), tokens, cost, latency and finish reason, so the `raw.jsonl` row matches the live one apart from `cached` and `cached_rounds`; a multi-round example counts as cached only when every round was a hit. A row with any round served from the cache (`cached_rounds > 0`) carries latency recorded on an earlier run, so it is left out of the report's live latency figures and its latency delta is marked not comparable. Errors are never cached (the next run re-sends a failed round; the rounds before it come from the cache), truncated responses are cached and stay flagged, and a hit records the original call's cost and latency. A `cache.db` written by an earlier version is upgraded in place on open and keeps its entries; several `evalshift` processes may open it at once. Disable per-project with `defaults.cache: false`; wipe with `evalshift cache clear`. +Live run-stage responses are cached in SQLite at `.evalshift/cache.db` in the folder the command runs in (beside `.evalshift/runs/`), so two projects never replay each other's answers. Entries are keyed by SHA-256 over canonical JSON of `(model, prompt, inputs, temperature, max_tokens[, history])` — plus `generation_config`, the tools array as sent, the round index and the sample index when each is set — with a 7-day TTL. Re-running an identical evaluation costs no run-stage calls, agent suites included. An example with a non-empty toolset gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, **the exact message list that round sends** (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and `strict` — reordering the tools is a miss), `generation_config` (so `tool_choice` and `parallel_tool_calls`), the effective temperature and `max_tokens`, the round index and the sample index — editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (calls with their ids and arguments, final text, refusal), tokens, cost, latency and finish reason, so the `raw.jsonl` row matches the live one apart from `cached` and `cached_rounds`; a multi-round example counts as cached only when every round was a hit. A row with any round served from the cache (`cached_rounds > 0`) carries latency recorded on an earlier run, so it is left out of the report's live latency figures and its latency delta is marked not comparable. Errors are never cached (the next run re-sends a failed round; the rounds before it come from the cache), truncated responses are cached and stay flagged, and a hit records the original call's cost and latency. A `cache.db` written by an earlier version is upgraded in place on open and keeps its entries; several `evalshift` processes may open it at once. A replayed call never reached the model, so it cannot show a model that changed behind the same name (another server behind the same id, a system prompt injected server-side, re-pulled weights): when any calls were replayed, `run` prints `⚠ N model calls were replayed from this project's response cache (T on the target) …` under its summary, and `compare` prints the same line above the verdict. `run --no-cache` / `compare --no-cache` send every run-stage call live and still write the fresh responses, replacing the cached ones, so the next ordinary run replays the new answers rather than the bypassed ones; the evaluate stage's judge and embedding cache is unaffected. Disable per-project with `defaults.cache: false` (no reads, no writes); wipe with `evalshift cache clear`. EvalShift 1.6.2 and earlier kept one machine-wide cache at `~/.evalshift/cache.db`; nothing reads it now, `cache clear` points at it when it exists, and it can be deleted. The cache covers the evaluate stage too: `semantic` embeddings are keyed by `(embedding model, text)`, and `llm_judge` verdicts by `(judge model, criterion, source output, target output)`. The judge key uses a canonical A/B ordering, so the per-call orientation randomization doesn't halve the hit rate — the orientation that was actually used is recorded with the verdict and replayed on a hit, leaving `metadata.target_was_a` faithful. @@ -912,7 +912,8 @@ Without `--ci`, warns after writing when an existing workflow under `.github/wor **`evalshift doctor`** — environment/config check. Exit 1 only on an invalid existing config or a blocking `semantic` / `llm_judge` model with no API key. Row 2, `evalshift-sdk`, confirms `import evalshift` is the SDK (`warn` when missing or shadowed, never a failure). Reports the toolset each configured suite carries and flags a suite whose examples carry more than one distinct toolset. The suite-side checks cover every suite in the config's `suites:` block, falling back to `./golden.jsonl` when none are wired. Adds a `ci pin` row when a workflow uses the GitHub Action (`warn` on pin drift, never a failure) and a `judge family` row when an `llm_judge` judge shares a provider with a configured arm (`warn`, never a failure). Adds a `captures.store` row when the config names a store (`warn` naming the package to install when the client library is missing, `warn` when the bucket cannot be listed), and an `evaluator keys` row when the config has a `semantic` or `llm_judge` evaluator (`fail` for a blocking one whose model has no key, `warn` for an advisory one, `ok` otherwise; top-level block and every named suite). Adds a `captures limits` row (`warn`) when `captures.store` is set without `captures.max_cases`, and, with either capture limit set, a `suite freshness` row per `source: captured` suite (`ok`, or `warn` per finding; never a failure). **`evalshift run`** — paired evaluation run (costs money — calls real models). -`-f/--from ` · `-t/--to ` · `-c/--config` · `-s/--suite ` · `--suite-name ` · `--resume` · `-y/--yes` +`-f/--from ` · `-t/--to ` · `-c/--config` · `-s/--suite ` · `--suite-name ` · `--resume` · `-y/--yes` · `--no-cache` (send every call live; fresh responses replace the cached ones — see [Response cache](#response-cache)) +Prints a `⚠` line under the summary when any calls were replayed from the cache. **`evalshift evaluate `** — score all pairs → `scores.jsonl`. `-c/--config` Prints a red doctor-style **broken eval harness** row when the *source* model failed the recorded ground truth on at least half of at least four `tool_selection.conformance` rows — the suite's expectations were captured from the source model, so the source is the one side that should satisfy them, and when it does not the run measured the harness rather than the migration. It names the rate (`10 of 10 … (100%)`) and the likely causes. `evalshift compare` prints the same row immediately above the verdict block. Below four conformance rows the check stays silent: a 100% rate over three has a 95% Wilson lower bound of 0.44 and cannot support the claim. @@ -925,7 +926,7 @@ On an interactive terminal, `report` and `compare` end with one line asking for **`evalshift compare`** — full pipeline under one live display: two models, **one** suite. It takes a single `--suite-name`/`--suite`, auto-selects when exactly one suite is wired, and prints a ready-to-run command per suite when several are. It does not run every suite — loop over the names, or use one CI job per suite. Formerly `all`; that name is hidden but still works, since scaffolded `EVALSHIFT.md` files reference it. All `run` flags, plus `--gate` · `--policy-gate` · `--open` · `--push` · `--insights/--no-insights` -With a capture limit set, prints the `suite freshness` findings for a named or auto-selected `source: captured` suite as `⚠` lines above the verdict. +With a capture limit set, prints the `suite freshness` findings for a named or auto-selected `source: captured` suite as `⚠` lines above the verdict. When any run-stage calls were replayed from the cache, prints the replay `⚠` line above the verdict too. ### Hosted @@ -955,7 +956,7 @@ With `captures.max_age` or `captures.max_cases` set, promotes only captures reco ### Housekeeping **`evalshift runs clean`** — `--keep ` · `--older-than ` · `--suite ` · `--dry-run` · `-y/--yes` · `--config` -**`evalshift cache clear`** — wipe the response cache. +**`evalshift cache clear`** — wipe this project's response cache (`.evalshift/cache.db` in the current folder) and name the path it cleared. When the old machine-wide `~/.evalshift/cache.db` exists it says so; it never deletes it. ### Hidden debug commands diff --git a/docs/configuration.md b/docs/configuration.md index 41451d97..58d6acd2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -417,7 +417,7 @@ A list of prompt definitions. Each entry has: | `judge_model` | string | `gemini-3.1-flash-lite-preview` | Default LLM-as-judge model. | | `insights_model`| string | (none) | Model that writes the run-insights narrative rendered in `report.html` and uploaded with the bundle. Falls back to `judge_model` when unset — writing analytical prose is a harder task than a pairwise A/B verdict, so it is worth tuning separately. See [Run insights](#run-insights). | | `concurrency` | int | 10 (1 ≤ x ≤ 64) | Max in-flight LLM calls during `evalshift run` **and** `evalshift evaluate` (the embedding and judge calls made while scoring). | -| `cache` | bool | `true` | Read/write the local SQLite cache at `~/.evalshift/cache.db`. Covers run-stage completions — for examples that offer tools, one entry per replayed round — plus `semantic` embeddings and `llm_judge` verdicts. See [Response cache](https://github.com/evalshift/evalshift-cli/blob/main/DOCS.md#response-cache) for the key. | +| `cache` | bool | `true` | Read/write the project's SQLite cache at `.evalshift/cache.db` (in the folder the command runs in). `run`/`compare --no-cache` bypass reads for one run while still writing. Covers run-stage completions — for examples that offer tools, one entry per replayed round — plus `semantic` embeddings and `llm_judge` verdicts. See [Response cache](https://github.com/evalshift/evalshift-cli/blob/main/DOCS.md#response-cache) for the key. | | `max_cost_usd` | float | 50.0 | Soft ceiling reserved for future enforcement. The pre-flight cost prompt currently triggers above $10 (skip with `--yes`). | | `max_tokens` | int | 4096 (`> 0`) | Completion length cap sent to every model call. Raise it if outputs are being truncated (the provider returns `finish_reason == "length"`); a `prompts[].max_tokens` entry overrides it per prompt. Truncated calls are detected, surfaced in the report, and **excluded from the regression statistics** so a cut-off output can't manufacture a false regression. | | `samples_per_example` | int | 1 (1 ≤ x ≤ 20) | How many times each `(prompt, example)` is sent to **each** model. Above 1, every sample is its own live call (the cache keys on the sample index), sample *i* of the source is scored against sample *i* of the target, and the example's row in `scores.jsonl` becomes the **mean over samples** with the per-sample scores and the within-example `delta_variance` under `metadata.samples`. The paired tests still run over examples, not samples, so this reduces noise without inflating `n`. Cost and the call count multiply by it; only worth turning on for a model that samples non-deterministically (see the report banner). See [Methodology](methodology.md#limitations-to-be-aware-of). | diff --git a/docs/faq.md b/docs/faq.md index 2fef303d..0aceea9e 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -7,7 +7,7 @@ machine to the LLM provider you configured — any provider LiteLLM supports — using your own API keys. -The local SQLite cache at `~/.evalshift/cache.db` only contains +The project's SQLite cache at `.evalshift/cache.db` only contains provider responses for *your* prompts and inputs. Hosted uploads are explicit. `bundle` packages the completed diff --git a/docs/getting-started.md b/docs/getting-started.md index 4aefc466..663edc01 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -38,8 +38,9 @@ fastest. EvalShift calls the provider you configure — any provider LiteLLM supports — directly using your own keys. Local runs do not send prompts or outputs to an EvalShift-operated server. -Provider responses are cached locally in `~/.evalshift/cache.db`, so re-running an -unchanged suite (agent suites included) makes no new calls. +Provider responses are cached in the project at `.evalshift/cache.db`, so re-running an +unchanged suite (agent suites included) makes no new calls. The run says how many calls it +replayed; pass `--no-cache` to send them all live. Set whichever providers you intend to use: diff --git a/docs/index.md b/docs/index.md index a1b8d630..d0eda814 100644 --- a/docs/index.md +++ b/docs/index.md @@ -90,7 +90,7 @@ Housekeeping: ``` evalshift inspect [--failed] # per-row deltas (--failed: negative or errored only) evalshift bundle -o out.json.gz # write the bundle somewhere else -evalshift cache clear # wipe the local response cache +evalshift cache clear # wipe this project's response cache evalshift runs clean # prune old runs (--keep, --older-than, --dry-run) ``` diff --git a/llms-full.txt b/llms-full.txt index 022ea1c8..5102c266 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -101,7 +101,8 @@ as one section under the pipeline block (also on stage failure); errors are neve | traces.jsonl | traces import | optional BYO agent traces | | run_bundle.json.gz | bundle/push | optional hosted upload bundle | -Cache: SQLite ~/.evalshift/cache.db, key = SHA-256 of canonical JSON {model, prompt, inputs, +Cache: SQLite .evalshift/cache.db in the folder the command runs in (beside .evalshift/runs/; +projects never share it), key = SHA-256 of canonical JSON {model, prompt, inputs, temperature, max_tokens[, history]} + generation_config, tools array as sent, round index, sample index when set; TTL 7 days. Tool-calling examples ARE cached, one entry per replayed round, keyed on canonical model + prompt + inputs + the exact message list that round sends (history, current @@ -115,7 +116,15 @@ cached_rounds > 0 = latency partly from an earlier run: excluded from live laten latency_comparable false. Errors never cached (next run re-sends the failed round; earlier rounds hit); truncated responses cached and still flagged; a hit records the original cost and latency. An older cache.db is upgraded in place and keeps its entries; concurrent opens by several -processes are safe. defaults.cache: false disables; `evalshift cache clear` wipes. +processes are safe. A replayed call never reached the model, so it cannot show a model changed +behind the same name (other server, server-side system prompt, re-pulled weights): when any +call was replayed, `run` prints "⚠ N model calls were replayed from this project's response cache +(T on the target) ..." under its summary and `compare` prints it above the verdict. +`run/compare --no-cache` sends every run-stage call live AND writes the fresh responses (replacing +the cached ones), so the next ordinary run replays the new answers; evaluate-stage judge/embedding +cache unaffected. defaults.cache: false disables (no reads, no writes); `evalshift cache clear` +wipes this project's cache. 1.6.2 and earlier used one machine-wide ~/.evalshift/cache.db; nothing +reads it now, `cache clear` mentions it when present (never deletes it). Resume: `run --resume` continues newest in_progress run; requires config_hash match (canonical config + suite PATH; suite contents are NOT hashed, so start fresh after editing examples), skips (prompt_id, example_id, role, sample_index) keys already in raw.jsonl; errored calls @@ -225,7 +234,7 @@ evalshift doctor "alpha: N case file(s) could not be read; the counts leave them out" Measured against defaults.source_model, models matched on bare names (see capture sync). evalshift run [-f/--from MODEL] [-t/--to MODEL] [-c CONFIG] [-s/--suite FILE] - [--suite-name NAME] [--resume] [-y/--yes] + [--suite-name NAME] [--resume] [-y/--yes] [--no-cache] Paired run over (prompt x example x {source,target}). --from/--to override defaults.source_model/target_model. --suite default ./golden.jsonl; --suite-name selects a key under suites: in config. Costs money — always calls real models. @@ -701,7 +710,7 @@ evalshift diff case RUN_ID EXAMPLE_ID (trace diff when traces exist, el evalshift replay case RUN_ID EXAMPLE_ID [--model source|target (default target)] [--trace] evalshift runs clean [--keep N] [--older-than DAYS] [--suite SLUG] [--dry-run] [-y] [--config] Precedence for keep count: --keep > EVALSHIFT_MAX_RUNS > config (default 20). 0 disables. -evalshift cache clear +evalshift cache clear (this project's .evalshift/cache.db) Hidden debug commands: evalshift validate [-s SUITE=golden.jsonl] [-c CONFIG] (config+suite+prompt cross-check; diff --git a/src/evalshift_cli/cache/schema.py b/src/evalshift_cli/cache/schema.py index fe3a766c..93d3e364 100644 --- a/src/evalshift_cli/cache/schema.py +++ b/src/evalshift_cli/cache/schema.py @@ -7,9 +7,10 @@ similarity searches; the simplest thing that could possibly work, per the PDF's guidance for the MVP. -The cache lives at ``~/.evalshift/cache.db`` by default. Tests pass an -in-memory SQLite URL via the ``database_url`` parameter on the engine -factory; never reach for the real DB in unit tests. +The cache lives at ``.evalshift/cache.db`` in the folder a command runs in, +beside ``.evalshift/runs/``, so two projects never replay each other's +responses. Tests pass an in-memory SQLite URL via the ``database_url`` +parameter on the engine factory; never reach for the real DB in unit tests. """ from __future__ import annotations @@ -21,7 +22,16 @@ from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column -DEFAULT_CACHE_PATH: Path = Path.home() / ".evalshift" / "cache.db" +DEFAULT_CACHE_PATH: Path = Path(".evalshift") / "cache.db" + + +def legacy_cache_path() -> Path: + """Where every project shared one cache before it moved into the project. + + Nothing reads it any more; ``evalshift cache clear`` points at it so it can + be deleted. A function rather than a constant so it follows ``$HOME``. + """ + return Path.home() / ".evalshift" / "cache.db" def _utcnow() -> datetime: @@ -112,4 +122,5 @@ def create_engine(database_url: str | None = None) -> AsyncEngine: "CachedCall", "create_engine", "default_database_url", + "legacy_cache_path", ] diff --git a/src/evalshift_cli/cache/store.py b/src/evalshift_cli/cache/store.py index db605f34..5ffbc9c3 100644 --- a/src/evalshift_cli/cache/store.py +++ b/src/evalshift_cli/cache/store.py @@ -158,6 +158,10 @@ class CacheStore: and one session's rollback-on-return would discard another's uncommitted write. File-backed databases use a real connection pool and run unserialised. + + A *refreshing* store (``refresh=True``, behind ``--no-cache``) misses on + every :meth:`get` but still writes on :meth:`put`, so the fresh responses + replace the ones it bypassed instead of leaving them for the next run. """ def __init__( @@ -165,10 +169,12 @@ def __init__( engine: AsyncEngine, *, ttl: timedelta = timedelta(days=DEFAULT_TTL_DAYS), + refresh: bool = False, ) -> None: self._engine = engine self._sessionmaker = async_sessionmaker(engine, expire_on_commit=False) self._ttl = ttl + self._refresh = refresh self._shared_connection_lock: asyncio.Lock | None = ( asyncio.Lock() if _pool_shares_one_connection(engine) else None ) @@ -180,6 +186,7 @@ async def open( *, ttl: timedelta = timedelta(days=DEFAULT_TTL_DAYS), path: Path | None = None, + refresh: bool = False, ) -> CacheStore: """Open the cache, creating the schema if necessary. @@ -188,6 +195,7 @@ async def open( ttl: How long cached entries are considered fresh. path: Override the on-disk location used when ``database_url`` is ``None``. + refresh: Never serve a hit; keep writing. See the class docstring. """ url = database_url or default_database_url(path) engine = create_engine(url) @@ -197,7 +205,7 @@ async def open( # No store owns the engine yet, so nothing else would close it. await engine.dispose() raise - return cls(engine, ttl=ttl) + return cls(engine, ttl=ttl, refresh=refresh) async def close(self) -> None: """Dispose of the underlying engine.""" @@ -221,7 +229,12 @@ async def _session(self) -> AsyncIterator[AsyncSession]: yield session async def get(self, key: str) -> CachedResponse | None: - """Return the cached response for ``key`` or ``None`` on miss/expiry.""" + """Return the cached response for ``key`` or ``None`` on miss/expiry. + + Always ``None`` on a refreshing store. + """ + if self._refresh: + return None cutoff = _utcnow() - self._ttl async with self._session() as session: stmt = select(CachedCall).where(CachedCall.cache_key == key) diff --git a/src/evalshift_cli/cli/commands/cache.py b/src/evalshift_cli/cli/commands/cache.py index 055075a5..ec60534e 100644 --- a/src/evalshift_cli/cli/commands/cache.py +++ b/src/evalshift_cli/cli/commands/cache.py @@ -1,6 +1,7 @@ """Implementation of ``evalshift cache clear``. -Tiny utility that deletes every row from the local SQLite cache. We +Tiny utility that deletes every row from this project's SQLite cache +(``.evalshift/cache.db`` in the current folder). We group cache subcommands under a Typer sub-app so future operations (``cache info``, ``cache prune``, …) slot in naturally without polluting the top-level command list. @@ -13,11 +14,12 @@ import typer from rich.console import Console +from evalshift_cli.cache import schema from evalshift_cli.cache.store import CacheStore cache_app = typer.Typer( name="cache", - help="Manage the local LLM-response cache.", + help="Manage this project's LLM-response cache.", no_args_is_help=True, add_completion=False, ) @@ -25,10 +27,18 @@ @cache_app.command(name="clear") def cache_clear() -> None: - """Remove every entry from the local cache.""" + """Remove every entry from this project's cache (.evalshift/cache.db).""" console = Console() removed = asyncio.run(_clear()) - console.print(f"[green]✓[/green] cleared {removed} cached call(s)") + console.print( + f"[green]✓[/green] cleared {removed} cached call(s) from {schema.DEFAULT_CACHE_PATH}" + ) + legacy = schema.legacy_cache_path() + if legacy.is_file(): + console.print( + f"[dim]{legacy} is the old machine-wide cache and is no longer used; " + "delete it to free the space.[/dim]" + ) async def _clear() -> int: diff --git a/src/evalshift_cli/cli/commands/compare.py b/src/evalshift_cli/cli/commands/compare.py index 9336b0a8..68c6fb02 100644 --- a/src/evalshift_cli/cli/commands/compare.py +++ b/src/evalshift_cli/cli/commands/compare.py @@ -65,6 +65,7 @@ run_evaluate, ) from evalshift_cli.cli.commands.report import run_report +from evalshift_cli.cli.commands.run import cache_replay_notice from evalshift_cli.config.loader import ConfigError, load_config from evalshift_cli.config.models import EvalShiftConfig, EvaluatorsConfig from evalshift_cli.evaluators.keys import LLM_JUDGE_KIND, SEMANTIC_EVALUATOR_NAME, EvaluatorKeyGap @@ -496,6 +497,16 @@ def compare_command( bool, typer.Option("--push", help="Push the completed run bundle to hosted EvalShift."), ] = False, + no_cache: Annotated[ + bool, + typer.Option( + "--no-cache", + help=( + "Send every model call live instead of replaying cached responses. " + "The fresh responses replace the cached ones." + ), + ), + ] = False, runs_base: Annotated[ Path, typer.Option("--runs-base", help="Base directory for run state (advanced).", hidden=True), @@ -679,6 +690,7 @@ def _on_progress(ev: ProgressEvent) -> None: suite_name=suite_name, console=console, on_progress=_on_progress, + no_cache=no_cache, ), ) except ( @@ -783,6 +795,12 @@ def _on_progress(ev: ProgressEvent) -> None: for line in freshness: console.print(line, soft_wrap=True) console.print() + # And again: a replayed call never reached the model, so the verdict below + # cannot see a model that changed behind the same name. + replay_notice = cache_replay_notice(run_result) + if replay_notice is not None: + console.print(replay_notice) + console.print() collapsed_note = format_collapsed_slices(analyze_result.collapsed_slices) if collapsed_note is not None: console.print(collapsed_note, style="dim") diff --git a/src/evalshift_cli/cli/commands/run.py b/src/evalshift_cli/cli/commands/run.py index 565c2800..ed76b8ad 100644 --- a/src/evalshift_cli/cli/commands/run.py +++ b/src/evalshift_cli/cli/commands/run.py @@ -109,6 +109,16 @@ def run( ), ), ] = False, + no_cache: Annotated[ + bool, + typer.Option( + "--no-cache", + help=( + "Send every model call live instead of replaying cached responses. " + "The fresh responses replace the cached ones." + ), + ), + ] = False, ) -> None: """Run paired evaluation on two models against your golden suite.""" console = Console() @@ -170,6 +180,7 @@ def run( run_slug=derive_suite_slug(suite_name=suite_name, suite_path=resolved_suite_path), suite_name=suite_name, console=console, + no_cache=no_cache, ), ) except PromptParseError as exc: @@ -192,6 +203,27 @@ def run( raise typer.Exit(code=1) from exc _print_summary(console, result) + notice = cache_replay_notice(result) + if notice is not None: + console.print(notice) + + +def cache_replay_notice(result: RunResult) -> str | None: + """The warning for a run that served calls from the response cache. + + A replayed call never reached the model, so a model that changed behind + the same name (another server, a server-side system prompt, re-pulled + weights) cannot show up in it. ``None`` when every call went out live. + """ + if result.cached_calls == 0: + return None + calls = "call was" if result.cached_calls == 1 else "calls were" + return ( + f"[yellow]⚠[/yellow] {result.cached_calls} model {calls} replayed from this " + f"project's response cache ({result.cached_target_calls} on the target) " + "instead of being sent. A model that changed behind the same name will not " + "show up in them; re-run with [bold]--no-cache[/bold] to send every call live." + ) def _print_summary(console: Console, result: RunResult) -> None: @@ -252,4 +284,4 @@ def _print_missing_api_keys( console.print("\n".join(lines)) -__all__ = ["run"] +__all__ = ["cache_replay_notice", "run"] diff --git a/src/evalshift_cli/runner/orchestrator.py b/src/evalshift_cli/runner/orchestrator.py index d55a9f40..216891e4 100644 --- a/src/evalshift_cli/runner/orchestrator.py +++ b/src/evalshift_cli/runner/orchestrator.py @@ -141,6 +141,7 @@ class RunResult: total_calls: int completed_calls: int cached_calls: int + cached_target_calls: int live_calls: int failed_calls: int total_cost_usd: float @@ -195,6 +196,7 @@ async def run_orchestrator( client: ModelClient | None = None, cache: CacheStore | None = None, on_progress: Callable[[ProgressEvent], None] | None = None, + no_cache: bool = False, ) -> RunResult: """Execute (or resume) a run end-to-end. @@ -219,6 +221,9 @@ async def run_orchestrator( console: Rich console (created if not supplied). client: :class:`ModelClient` to use; mostly here for tests. cache: :class:`CacheStore` to use; mostly here for tests. + no_cache: ``--no-cache``: send every call live, still caching the + fresh responses. Applies to the cache this function opens; a + ``cache`` passed in is used as given. """ cons = console or Console() @@ -304,7 +309,7 @@ async def run_orchestrator( # Real work. cache_owned = cache is None - cache_inst = cache or await CacheStore.open() + cache_inst = cache or await CacheStore.open(refresh=no_cache) client_inst = client or ModelClient() try: @@ -1074,6 +1079,7 @@ async def _process_work( write_lock = asyncio.Lock() completed = already_done cached_count = 0 + cached_target_count = 0 live_count = 0 failed_count = 0 total_cost = 0.0 @@ -1103,7 +1109,8 @@ async def _process_work( template_by_id: dict[str, PromptTemplate] = {t.id: t for t in templates} async def _do_one(item: WorkItem) -> None: - nonlocal completed, cached_count, live_count, failed_count, total_cost + nonlocal completed, cached_count, cached_target_count, live_count, failed_count + nonlocal total_cost async with sem: tmpl = template_by_id[item.prompt.id] prompt_text = render(tmpl.content, item.example.inputs) @@ -1122,6 +1129,8 @@ async def _do_one(item: WorkItem) -> None: completed += 1 if call.cached: cached_count += 1 + if call.role == "target": + cached_target_count += 1 elif call.error is None: live_count += 1 if call.error is not None: @@ -1174,6 +1183,7 @@ async def _do_one(item: WorkItem) -> None: total_calls=total, completed_calls=completed, cached_calls=cached_count, + cached_target_calls=cached_target_count, live_calls=live_count, failed_calls=failed_count, total_cost_usd=total_cost, diff --git a/tests/conftest.py b/tests/conftest.py index 1af7f9e2..b9908694 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -349,8 +349,9 @@ def _isolated_response_cache( Command-level tests (``run``, ``compare``, ``evaluate`` and the integration pipelines) open the cache without passing a URL, which resolves to - ``~/.evalshift/cache.db``. Unredirected, they wrote the developer's real - cache, and a row left there by one test run could serve the next one. + ``.evalshift/cache.db`` under the current directory — the repo root when + a test does not ``chdir``. Unredirected, a row left there by one test run + could serve the next one. ``tmp_path_factory`` rather than ``tmp_path``, so tests that inspect their own ``tmp_path`` never find a stray ``cache.db`` in it. """ diff --git a/tests/unit/test_cache.py b/tests/unit/test_cache.py index fca19986..7e4cef03 100644 --- a/tests/unit/test_cache.py +++ b/tests/unit/test_cache.py @@ -22,6 +22,7 @@ from sqlalchemy.ext.asyncio import AsyncEngine from typer.testing import CliRunner +from evalshift_cli.cache.schema import DEFAULT_CACHE_PATH as SHIPPED_CACHE_PATH from evalshift_cli.cache.schema import Base, create_engine from evalshift_cli.cache.store import CacheStore, cache_key from evalshift_cli.cli.main import app @@ -30,6 +31,9 @@ # Use an in-memory database for every test to keep them fast and hermetic. IN_MEMORY_DB = "sqlite+aiosqlite:///:memory:" +# SHIPPED_CACHE_PATH is bound at import, before the autouse fixture in +# tests/conftest.py redirects the module attribute, so it is the real default. + # --------------------------------------------------------------------------- # cache_key — pure function @@ -302,8 +306,8 @@ def test_no_tools_payload_is_byte_identical_to_pre_toolset_payload(self) -> None "temperature": temperature, "max_tokens": max_tokens, } - assert cache_key(**base) == expected # type: ignore[arg-type] - assert cache_key(**base, tools_payload=None) == expected # type: ignore[arg-type] + assert cache_key(**base) == expected + assert cache_key(**base, tools_payload=None) == expected def _tool_key(self, tools_payload: list[dict[str, Any]] | None) -> str: return cache_key( @@ -529,7 +533,7 @@ async def test_concurrent_puts_of_one_key_do_not_collide(self, store: CacheStore import asyncio kw = _put_kwargs() - await asyncio.gather(*(store.put("k", **kw) for _ in range(4))) # type: ignore[arg-type] + await asyncio.gather(*(store.put("k", **kw) for _ in range(4))) got = await store.get("k") assert got is not None assert got.response_text == "Hello Alex!" @@ -592,7 +596,7 @@ async def test_concurrent_misses_on_distinct_keys_all_persist(self) -> None: async def miss_then_put(key: str) -> None: assert await store.get(key) is None - await store.put(key, **_put_kwargs()) # type: ignore[arg-type] + await store.put(key, **_put_kwargs()) try: await asyncio.gather(*(miss_then_put(k) for k in keys)) @@ -624,7 +628,7 @@ def on_checkin(*_: object) -> None: event.listen(pool, "checkin", on_checkin) try: await asyncio.gather( - *(store.put(f"k{i}", **_put_kwargs()) for i in range(4)), # type: ignore[arg-type] + *(store.put(f"k{i}", **_put_kwargs()) for i in range(4)), *(store.get(f"k{i}") for i in range(4)), store.count(), ) @@ -647,8 +651,8 @@ class TestCacheClearCommand: def test_clear_runs_against_fresh_db( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: - # Redirect the default cache path into tmp so we don't touch the - # user's real ~/.evalshift/cache.db. + # Redirect the default cache path into tmp so we never clear the + # repo's own .evalshift/cache.db. monkeypatch.setattr( "evalshift_cli.cache.schema.DEFAULT_CACHE_PATH", tmp_path / "cache.db", @@ -657,6 +661,52 @@ def test_clear_runs_against_fresh_db( assert result.exit_code == 0, result.stdout assert "cleared" in result.stdout + def test_clear_empties_this_projects_cache( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setattr("evalshift_cli.cache.schema.DEFAULT_CACHE_PATH", SHIPPED_CACHE_PATH) + monkeypatch.chdir(tmp_path) + + async def seed() -> None: + store = await CacheStore.open() + try: + await store.put("k", **_put_kwargs()) + finally: + await store.close() + + asyncio.run(seed()) + result = runner.invoke(app, ["cache", "clear"]) + assert result.exit_code == 0, result.stdout + assert "cleared 1 cached call(s)" in result.stdout + assert ".evalshift/cache.db" in result.stdout + + def test_clear_points_at_the_old_machine_wide_cache( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Before the cache moved into the project, every project shared + # ~/.evalshift/cache.db. Nothing reads it any more; clear says where it + # is so it can be deleted, but never deletes it itself. + home = tmp_path / "home" + legacy = home / ".evalshift" / "cache.db" + legacy.parent.mkdir(parents=True) + legacy.write_bytes(b"") + monkeypatch.setenv("HOME", str(home)) + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["cache", "clear"]) + assert result.exit_code == 0, result.stdout + flat = " ".join(result.stdout.split()) + assert "no longer used" in flat + assert legacy.is_file() + + def test_clear_is_quiet_without_an_old_cache( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + monkeypatch.setenv("HOME", str(tmp_path / "home")) + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["cache", "clear"]) + assert result.exit_code == 0, result.stdout + assert "no longer used" not in result.stdout + def test_cache_help_lists_clear(self) -> None: result = runner.invoke(app, ["cache", "--help"]) assert "clear" in result.stdout @@ -740,7 +790,7 @@ class TestCacheStoreToolTrace: async def test_round_trip_preserves_the_trace_exactly(self, store: CacheStore) -> None: trace = _rich_trace() kw = _tool_put_kwargs(trace) - await store.put("k", **kw, finish_reason="tool_calls", trace=trace) # type: ignore[arg-type] + await store.put("k", **kw, finish_reason="tool_calls", trace=trace) got = await store.get("k") assert got is not None assert got.trace == trace @@ -756,7 +806,7 @@ async def test_tool_only_trace_round_trips(self, store: CacheStore) -> None: calls=[ToolCall(tool_name="t", arguments={}, call_id="c1", sequence_index=0)], final_text=None, ) - await store.put("k", **_tool_put_kwargs(trace), trace=trace) # type: ignore[arg-type] + await store.put("k", **_tool_put_kwargs(trace), trace=trace) got = await store.get("k") assert got is not None assert got.trace == trace @@ -764,7 +814,7 @@ async def test_tool_only_trace_round_trips(self, store: CacheStore) -> None: assert got.trace.final_text is None async def test_text_rows_carry_no_trace(self, store: CacheStore) -> None: - await store.put("k", **_put_kwargs()) # type: ignore[arg-type] + await store.put("k", **_put_kwargs()) got = await store.get("k") assert got is not None assert got.trace is None @@ -776,7 +826,7 @@ async def test_a_trace_that_no_longer_validates_is_a_miss(self, tmp_path: Path) store = await CacheStore.open(database_url=url) try: trace = _rich_trace() - await store.put("k", **_tool_put_kwargs(trace), trace=trace) # type: ignore[arg-type] + await store.put("k", **_tool_put_kwargs(trace), trace=trace) async with store._engine.begin() as conn: await conn.execute( text("UPDATE cached_calls SET trace_json = :j"), @@ -846,7 +896,7 @@ async def test_tool_rows_can_be_written_after_the_backfill(self, tmp_path: Path) store = await CacheStore.open(database_url=url) try: trace = _rich_trace() - await store.put("new", **_tool_put_kwargs(trace), trace=trace) # type: ignore[arg-type] + await store.put("new", **_tool_put_kwargs(trace), trace=trace) got = await store.get("new") assert got is not None assert got.trace == trace @@ -953,12 +1003,90 @@ def test_reordering_the_tools_changes_the_key(self) -> None: def test_the_suite_never_opens_the_users_real_cache() -> None: # Several command-level tests open the default on-disk cache. Without the - # autouse redirect in tests/conftest.py they read and wrote the developer's - # ~/.evalshift/cache.db, so a cached row from an earlier test run could - # serve a later one. + # autouse redirect in tests/conftest.py they read and wrote whatever + # .evalshift/cache.db the working directory held, so a cached row from an + # earlier test run could serve a later one. from evalshift_cli.cache import schema - assert Path.home() / ".evalshift" / "cache.db" != schema.DEFAULT_CACHE_PATH + assert schema.DEFAULT_CACHE_PATH != SHIPPED_CACHE_PATH + assert schema.DEFAULT_CACHE_PATH.is_absolute() + + +class TestProjectScopedCache: + """The cache belongs to the folder a command runs in, beside ``.evalshift/runs/``. + + It used to live at ``~/.evalshift/cache.db`` and be shared by every project + on the machine, so a candidate re-pointed at another backend under the same + model name replayed another project's answers. + """ + + def test_the_default_is_relative_to_the_project(self) -> None: + assert Path(".evalshift") / "cache.db" == SHIPPED_CACHE_PATH + + async def test_two_project_folders_do_not_share_responses( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr("evalshift_cli.cache.schema.DEFAULT_CACHE_PATH", SHIPPED_CACHE_PATH) + first, second = tmp_path / "first", tmp_path / "second" + first.mkdir() + second.mkdir() + + monkeypatch.chdir(first) + store = await CacheStore.open() + try: + await store.put("k", **_put_kwargs()) + finally: + await store.close() + assert (first / ".evalshift" / "cache.db").is_file() + + monkeypatch.chdir(second) + store = await CacheStore.open() + try: + assert await store.get("k") is None + finally: + await store.close() + + +class TestRefresh: + """``refresh=True`` backs ``--no-cache``: every lookup misses, every write lands.""" + + async def test_a_refreshing_store_never_serves_a_hit(self, tmp_path: Path) -> None: + url = f"sqlite+aiosqlite:///{tmp_path / 'c.db'}" + plain = await CacheStore.open(database_url=url) + try: + await plain.put("k", **_put_kwargs()) + finally: + await plain.close() + + refreshing = await CacheStore.open(database_url=url, refresh=True) + try: + assert await refreshing.get("k") is None + finally: + await refreshing.close() + + async def test_its_writes_replace_the_stale_entry(self, tmp_path: Path) -> None: + # Otherwise the next run without --no-cache would replay the stale + # answer the user just bypassed. + url = f"sqlite+aiosqlite:///{tmp_path / 'c.db'}" + plain = await CacheStore.open(database_url=url) + try: + await plain.put("k", **_put_kwargs()) + finally: + await plain.close() + + refreshing = await CacheStore.open(database_url=url, refresh=True) + try: + await refreshing.put("k", **{**_put_kwargs(), "response_text": "fresh"}) + finally: + await refreshing.close() + + again = await CacheStore.open(database_url=url) + try: + hit = await again.get("k") + finally: + await again.close() + assert hit is not None + assert hit.response_text == "fresh" class TestConcurrentSchemaSetup: diff --git a/tests/unit/test_compare_command.py b/tests/unit/test_compare_command.py index b94069f3..98969e61 100644 --- a/tests/unit/test_compare_command.py +++ b/tests/unit/test_compare_command.py @@ -929,3 +929,36 @@ def test_marks_a_skipped_semantic_and_an_all_skipped_judge(self) -> None: gaps = evaluator_key_gaps(evaluators, {}) assert _evaluator_family_summary(evaluators, gaps) == "semantic (no key) · judge (no key)" assert _evaluator_family_summary(evaluators) == "semantic · judge" + + +class TestCacheReplay: + def test_a_replayed_compare_warns_above_the_verdict( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold(tmp_path) + _patch_client(monkeypatch) + monkeypatch.chdir(tmp_path) + first = runner.invoke(app, ["all", "--yes"]) + assert first.exit_code == 0, first.output + assert "replayed" not in first.output + + result = runner.invoke(app, ["all", "--yes"]) + + assert result.exit_code == 0, result.output + flat = " ".join(result.output.split()) + assert "8 model calls were replayed from this project's response cache" in flat + # Above the verdict block, which ends with the run id line. + assert flat.index("replayed") < flat.index("no significant change") + + def test_no_cache_drops_the_warning( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold(tmp_path) + _patch_client(monkeypatch) + monkeypatch.chdir(tmp_path) + runner.invoke(app, ["all", "--yes"]) + + result = runner.invoke(app, ["all", "--yes", "--no-cache"]) + + assert result.exit_code == 0, result.output + assert "replayed" not in result.output diff --git a/tests/unit/test_run_command.py b/tests/unit/test_run_command.py index c2030203..db66e993 100644 --- a/tests/unit/test_run_command.py +++ b/tests/unit/test_run_command.py @@ -8,6 +8,7 @@ from __future__ import annotations +import json from pathlib import Path from typing import Any @@ -369,3 +370,103 @@ def test_every_evaluator_skipped_exits_before_the_run( assert "every configured evaluator would be skipped" in flat assert "OPENAI_API_KEY" in flat assert not (tmp_path / ".evalshift" / "runs").exists() + + +# --------------------------------------------------------------------------- +# Response cache: replay notice and --no-cache +# --------------------------------------------------------------------------- + + +class _CountingClient: + """Fake ``ModelClient.complete`` that counts live calls and answers ``text``.""" + + def __init__(self, monkeypatch: pytest.MonkeyPatch) -> None: + self.calls = 0 + self.text = "ok" + + async def fake_complete(_client: ModelClient, **kwargs: Any) -> CompletionResult: + self.calls += 1 + return CompletionResult( + text=self.text, + model_id=str(kwargs["model"]), + input_tokens=5, + output_tokens=2, + cost_usd=0.0, + latency_ms=10, + ) + + monkeypatch.setattr(ModelClient, "complete", fake_complete) + _set_api_keys(monkeypatch) + + +def _flat(output: str) -> str: + return " ".join(output.split()) + + +class TestCacheReplay: + def test_a_fully_live_run_has_no_notice( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold(tmp_path) + _CountingClient(monkeypatch) + monkeypatch.chdir(tmp_path) + result = runner.invoke(app, ["run", "--yes"]) + assert result.exit_code == 0, result.stdout + assert "replayed" not in result.stdout + + def test_a_replayed_run_says_so_and_names_the_way_out( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold(tmp_path) + _CountingClient(monkeypatch) + monkeypatch.chdir(tmp_path) + runner.invoke(app, ["run", "--yes"]) + + result = runner.invoke(app, ["run", "--yes"]) + + assert result.exit_code == 0, result.stdout + flat = _flat(result.stdout) + # 2 examples x {source, target}, all served from the first run. + assert "4 model calls were replayed from this project's response cache" in flat + assert "2 on the target" in flat + assert "--no-cache" in flat + + def test_no_cache_sends_every_call_live( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold(tmp_path) + client = _CountingClient(monkeypatch) + monkeypatch.chdir(tmp_path) + runner.invoke(app, ["run", "--yes"]) + assert client.calls == 4 + + result = runner.invoke(app, ["run", "--yes", "--no-cache"]) + + assert result.exit_code == 0, result.stdout + assert client.calls == 8 + assert "replayed" not in result.stdout + + def test_the_next_cached_run_replays_the_fresh_answers( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # --no-cache still writes, so the stale answers it bypassed do not + # come back on the next ordinary run. + _scaffold(tmp_path) + client = _CountingClient(monkeypatch) + monkeypatch.chdir(tmp_path) + runner.invoke(app, ["run", "--yes"]) + client.text = "fresh" + runner.invoke(app, ["run", "--yes", "--no-cache"]) + before = set((tmp_path / ".evalshift" / "runs").iterdir()) + + result = runner.invoke(app, ["run", "--yes"]) + + assert result.exit_code == 0, result.stdout + assert client.calls == 8 + [latest] = set((tmp_path / ".evalshift" / "runs").iterdir()) - before + rows = [ + json.loads(line) + for line in (latest / "raw.jsonl").read_text(encoding="utf-8").splitlines() + ] + assert {row["text"] for row in rows} == {"fresh"} + assert all(row["cached"] for row in rows)