docs: clarify eval policy quickstart

README.md

@@ -20,6 +20,22 @@ The package is usable for deterministic local eval suites. Model-assisted comman
 but consumers must configure Promptfoo and calibrate judge behavior before treating model output as
 more than advisory evidence.
 
+## Evaluation policy
+
+Keep evals in three lanes:
+
+- **CI / `pnpm check`:** fast, offline, structural checks only. Suitable checks include
+  format/lint, static docs or schema validation, adapter import/syntax validation, fixture manifest
+  validation, local unit tests for graders/helpers, and seeded fixture checks that never call
+  external providers.
+- **Local on-demand:** deterministic suite runs, including semantic case portfolios, run before
+  significant changes in the consumer repo. These commands may create result bundles and should use
+  the consumer's documented local scripts.
+- **Manual/advisory:** Promptfoo/Codex model-assisted generation, pointwise judging, pairwise
+  judging, LLM judge coverage, long product-to-design or product-to-plan session evals, and
+  expensive replay suites. These require explicit local setup, auth where relevant, and human
+  calibration. They are not default CI gates.
+
 ## What it is
 
 `eval-kit` is a small runner and bootstrap package for repository-local evaluation suites.
@@ -67,23 +83,23 @@ Install from a Git tag in a consumer repo:
 Bootstrap a generic deterministic suite:
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
-eval-kit doctor
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
+pnpm exec eval-kit doctor
 ```
 
 Create a case:
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
-eval-kit list-cases
-eval-kit doctor
+pnpm exec eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit list-cases
+pnpm exec eval-kit doctor
 ```
 
 Run a deterministic case:
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --config evals/eval-kit.config.json \
   --case case-example-v1 \
   --candidate path/to/candidate.md \
@@ -160,7 +176,10 @@ pnpm install --frozen-lockfile
 pnpm check
 ```
 
-`pnpm check` is the required local and CI gate.
+`pnpm check` is the required local and CI gate for fast, offline, structural checks. Keep
+run-producing semantic portfolios, model-assisted generation/judging, LLM judge coverage, pairwise
+judging, and expensive full-case replays out of the default gate unless a consumer repo documents a
+narrow deterministic structural subset.
 
 ## Release model
 

docs/design/bootstrap-cli.md

@@ -18,8 +18,8 @@ eval-kit list-cases
 Creates a generic deterministic suite skeleton.
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
 ```
 
 Generated files:
@@ -47,7 +47,7 @@ Rules:
 Creates a generic deterministic case.
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit scaffold-case --case case-example-v1
 ```
 
 Generated files:
@@ -83,6 +83,10 @@ Checks:
 - consumer fixture validation hook passes;
 - Promptfoo exists only when model-assisted methods are enabled.
 
+Policy note: `doctor` reports suite validity, not CI policy compliance. Consumer default CI should
+stay fast and offline, without run-producing, model-assisted, networked, or manually calibrated
+evals.
+
 ## `list-cases`
 
 Lists discovered case IDs from the configured case manifests.

docs/design/consumer-integration.md

@@ -27,7 +27,12 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva
 }
 ```
 
-Consumers decide whether evals are part of `pnpm check`. Start with manual scripts until the suite is stable, then add deterministic checks to `pnpm check`.
+Consumers decide which eval checks belong in each lane. Keep `pnpm check` fast, offline, and
+structural: format/lint, static schema/docs validation, adapter import/syntax validation, fixture
+manifest validation, local grader/helper unit tests, and seeded fixture checks that do not call
+external providers. Run deterministic semantic case portfolios locally before significant changes.
+Keep Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long session evals, and
+expensive full-case replays manual/advisory rather than default CI gates.
 
 ## Consumer-owned files
 
@@ -50,9 +55,9 @@ evals/
 - [ ] Bootstrap or create evals/ files.
 - [ ] Replace generic adapter semantics with repo-local semantics.
 - [ ] Add at least one deterministic case.
-- [ ] Run eval-kit doctor.
+- [ ] Run `pnpm exec eval-kit doctor`.
 - [ ] Run one deterministic case.
-- [ ] Decide whether deterministic evals enter pnpm check.
+- [ ] Decide whether any fast, offline structural checks enter `pnpm check`.
 - [ ] Document local eval commands in README or evals/README.md.
 ```
 
@@ -63,3 +68,4 @@ evals/
 - Do not put suite semantics into eval-kit.
 - Do not enable Promptfoo commands without local dependency/auth setup.
 - Do not trust model judges without calibration.
+- Do not require auth, network, external providers, or manual calibration in `pnpm check`.

docs/design/skills.md

@@ -23,10 +23,10 @@ Must guide:
 
 - read repo instructions;
 - add dependency;
-- run `eval-kit init --suite generic --dry-run`;
+- run `pnpm exec eval-kit init --suite generic --dry-run`;
 - inspect planned files;
-- run `eval-kit init --suite generic`;
-- run `eval-kit doctor`;
+- run `pnpm exec eval-kit init --suite generic`;
+- run `pnpm exec eval-kit doctor`;
 - report remaining consumer decisions.
 
 Must not:
@@ -56,6 +56,7 @@ Must detect:
 - hidden requirements in reference outputs;
 - vague rubrics;
 - model judges used as hard gates without calibration;
+- run-producing semantic portfolios treated as default CI gates;
 - cases too large to debug;
 - consumer semantics leaking into eval-kit.
 

docs/design/technical-design-follow-up.md

@@ -43,14 +43,14 @@ In `technical-design/evals/adapter.mjs`:
 
 4. Add tests that call these adapter exports and assert required keys exist.
 
-5. Run:
+5. Run the fast/offline gate and local on-demand deterministic smoke:
 
 ```bash
 pnpm check
 pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit
 ```
 
-Optional if local auth exists:
+Manual/advisory if local auth exists:
 
 ```bash
 pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model <model> --provider openai --effort medium --run-id verify-generation-vars

docs/examples.md

@@ -21,7 +21,7 @@ evals/
 ## Deterministic Case
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --config evals/eval-kit.config.json \
   --case case-alpha \
   --candidate evals/cases/case-alpha/candidate.md \
@@ -85,7 +85,7 @@ export const renderReport = ({ caseId, grades, findings }) => {
 ## Fixture Validation
 
 ```bash
-eval-kit validate-fixtures --config evals/eval-kit.config.json
+pnpm exec eval-kit validate-fixtures --config evals/eval-kit.config.json
 ```
 
 The kit validates discovered case manifests first. If `adapter.validateFixtures` exists, the kit
@@ -129,9 +129,10 @@ await runCase({
 `generate`, `judge-coverage`, and `judge-pairwise` are optional Promptfoo-backed commands. They
 require a consumer adapter that returns the variables expected by the selected prompt template.
 Consumers can use the bundled generic prompts or set `prompt_templates` to suite-owned prompts.
+Treat these as manual/advisory runs, not default CI gates.
 
 ```bash
-eval-kit judge-coverage \
+pnpm exec eval-kit judge-coverage \
   --config evals/eval-kit.config.json \
   --case case-alpha \
   --candidate evals/cases/case-alpha/candidate.md \

docs/guides/consumer-integration.md

@@ -32,9 +32,9 @@ pnpm install
 ## Bootstrap generic files
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
-eval-kit doctor
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
+pnpm exec eval-kit doctor
 ```
 
 ## Replace semantics
@@ -53,7 +53,7 @@ evals/prompts/*            # local prompts, if overriding bundled generic prompt
 
 ## Decide gate level
 
-Start manual:
+Start with package scripts for local use:
 
 ```json
 {
@@ -64,9 +64,19 @@ Start manual:
 }
 ```
 
-Once stable, add deterministic evals to `pnpm check`.
+Use three lanes:
 
-Do not add model-assisted evals to `pnpm check` unless the repo has explicit policy for credentials, cost, determinism, and judge calibration.
+- **CI / `pnpm check`:** fast, offline, structural checks only. This can include format/lint,
+  static docs/schema validation, adapter import or syntax validation, fixture manifest validation,
+  local unit tests for graders/helpers, and seeded fixture checks that do not require auth, network,
+  model calls, or manual calibration.
+- **Local on-demand:** deterministic `run-case` suites and semantic case portfolios. Run these
+  before significant changes, using the consumer repo's scripts or `pnpm exec eval-kit`.
+- **Manual/advisory:** Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long
+  product-to-design/product-to-plan session evals, and expensive full-case replay suites.
+
+Do not add run-producing semantic portfolios or model-assisted evals to the default CI gate. If a
+consumer wants a deterministic subset in `pnpm check`, keep it short, offline, and structural.
 
 ## Review checklist
 
@@ -76,5 +86,5 @@ Do not add model-assisted evals to `pnpm check` unless the repo has explicit pol
 - [ ] Are hidden reference answers avoided?
 - [ ] Are deterministic blockers not averaged away?
 - [ ] Are model judges advisory unless calibrated?
-- [ ] Does pnpm check remain reasonable for contributors?
+- [ ] Does `pnpm check` remain fast, offline, and reasonable for contributors?
 ```

docs/guides/model-assisted-evals.md

@@ -1,6 +1,7 @@
 # Model-assisted evals
 
-Model-assisted evals are optional. Deterministic evals are the default.
+Model-assisted evals are optional manual/advisory checks. Deterministic evals are the default
+local evidence path, and model-assisted commands are never default CI gates.
 
 ## When to use
 
@@ -23,20 +24,20 @@ A consumer repo must provide:
 - adapter hooks for prompt variables;
 - rubrics;
 - result review process;
-- calibration policy before using judge results as gates.
+- calibration policy before using judge results as gates outside default CI.
 
 ## Commands
 
 ```bash
-eval-kit generate \
+pnpm exec eval-kit generate \
   --case <case-id> \
   --model <model> \
   --provider openai \
   --effort medium \
   --run-id <run-id>
 
 
-eval-kit judge-coverage \
+pnpm exec eval-kit judge-coverage \
   --case <case-id> \
   --candidate <candidate.md> \
   --model <model> \
@@ -45,7 +46,7 @@ eval-kit judge-coverage \
   --run-id <run-id>
 
 
-eval-kit judge-pairwise \
+pnpm exec eval-kit judge-pairwise \
   --case <case-id> \
   --candidate-a <a.md> \
   --candidate-b <b.md> \

docs/guides/quickstart.md

@@ -21,22 +21,22 @@ pnpm install
 ## 2. Bootstrap files
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
 ```
 
 ## 3. Validate setup
 
 ```bash
-eval-kit doctor
+pnpm exec eval-kit doctor
 ```
 
 ## 4. Create a case
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
-eval-kit list-cases
-eval-kit doctor
+pnpm exec eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit list-cases
+pnpm exec eval-kit doctor
 ```
 
 ## 5. Replace placeholders
@@ -57,14 +57,17 @@ The generated adapter checks for exact required text. That is a starter, not a p
 Create or choose a candidate file, then run:
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --case case-example-v1 \
   --candidate path/to/candidate.md \
   --run-id verify-example
 ```
 
 Results are written under the configured results root.
 
+Deterministic case runs are local on-demand evidence. Consumers should run the documented semantic
+case portfolio before significant changes, but run-producing evals are not default CI gates.
+
 ## 7. Add scripts
 
 Recommended scripts:
@@ -78,3 +81,6 @@ Recommended scripts:
   }
 }
 ```
+
+Package scripts may call the local binary as `eval-kit`. Interactive shell examples use
+`pnpm exec eval-kit` so consumers run the pinned dependency from their repo.

docs/guides/technical-design-integration-fix.md

@@ -13,7 +13,7 @@ This guide is for the follow-up PR in `agentic-workflow-kit/technical-design`.
 3. Inspect `evals/adapter.mjs`.
 4. Update generation, pointwise, and pairwise variable resolvers to include the generic keys expected by eval-kit prompts.
 5. Add unit tests for adapter variable contracts.
-6. Run the normal gate and deterministic smoke.
+6. Run the fast/offline `pnpm check` gate and the local on-demand deterministic smoke.
 
 ## Required keys
 
@@ -59,7 +59,7 @@ pnpm check
 pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit
 ```
 
-Optional if auth is available:
+Manual/advisory if auth is available:
 
 ```bash
 pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model <model> --provider openai --effort medium --run-id verify-generation-vars

docs/guides/writing-cases.md

@@ -58,6 +58,12 @@ Expected items should be:
 
 Rubrics are for semantic interpretation. They should not contain hidden hard requirements that are absent from source artifacts.
 
+## Case portfolios
+
+Semantic case portfolios are local on-demand evidence. Keep cases small enough to debug, then run
+the consumer repo's deterministic portfolio before significant changes. Do not make long
+run-producing portfolios, full-case replays, or model-assisted judging part of default CI.
+
 ## Bad signs
 
 Avoid cases where: