[babaliauskas]

Why

A tester found that a second run under the same candidate model name, from a different project folder, replayed the earlier answers. Answers only went live again once they renamed the candidate.

The cache was one machine-wide ~/.evalshift/cache.db, keyed on model name, prompt, inputs, temperature and max_tokens. Prompt and system-prompt edits that EvalShift sends are already in the key. Anything it doesn't send can't be in the key: another server behind the same id, a system prompt added server-side, or re-pulled weights. Those changes were invisible, and nothing told the user that calls had been replayed.

What

• The cache belongs to the project. It moved from ~/.evalshift/cache.db to .evalshift/cache.db in the folder the command runs in, next to .evalshift/runs/. evalshift cache clear names the path it cleared and mentions the old machine-wide file when it exists. It never deletes that file.
• run --no-cache / compare --no-cache. Every run-stage call is sent live, and the fresh responses still get written (CacheStore(refresh=True): reads always miss, writes still land). Without that, the next normal run would replay the answers the user just bypassed. The evaluate stage's judge and embedding cache is unaffected.
• Replay notice. When calls came from the cache, run prints ⚠ N model calls were replayed from this project's response cache (T on the target) … under its summary, and compare prints it above the verdict. RunResult gains cached_target_calls.

Docs travel with the change: DOCS.md, llms-full.txt, docs/configuration.md, docs/getting-started.md, docs/faq.md, docs/index.md, CHANGELOG.md (Added + Changed).

I chose not to add the endpoint (base URL) to the cache key, to avoid extra complexity for users.

Notes for reviewers

• The first run in each project starts with an empty cache. The CI action already starts cold on every run, so it isn't affected.
• The scaffolded .gitignore advice (.evalshift/*) already covers the new file.

Testing

• New tests: project isolation (two folders don't share entries), the refresh store (misses, still overwrites), cache clear path and legacy hint, and run/compare notice and --no-cache end to end. That includes the case "after --no-cache, the next cached run replays the fresh answers".
• ruff, ruff format, mypy --strict and pre-commit run --all-files pass.
• 3518 tests pass, 1 fails: tests/unit/test_cli_capture_remote.py::test_online_sync_says_it_ignores_limits_it_cannot_read. It fails on main too. It's flaky because the console wraps the temp folder path, and it's unrelated to this change.

🤖 Generated with Claude Code