From 75c027ec4787c185c32f5738fe3e957703b4c1b4 Mon Sep 17 00:00:00 2001
From: Arye Kogan <aryekogan@gmail.com>
Date: Fri, 3 Jul 2026 22:20:42 +0300
Subject: [PATCH] docs: clarify eval policy quickstart

---
 README.md                                     | 35 ++++++++++++++-----
 docs/design/bootstrap-cli.md                  | 10 ++++--
 docs/design/consumer-integration.md           | 12 +++++--
 docs/design/skills.md                         |  7 ++--
 docs/design/technical-design-follow-up.md     |  4 +--
 docs/examples.md                              |  7 ++--
 docs/guides/consumer-integration.md           | 24 +++++++++----
 docs/guides/model-assisted-evals.md           | 11 +++---
 docs/guides/quickstart.md                     | 20 +++++++----
 .../technical-design-integration-fix.md       |  4 +--
 docs/guides/writing-cases.md                  |  6 ++++
 docs/reference/cli.md                         | 27 +++++++-------
 docs/reference/config.md                      | 14 ++++++++
 docs/reference/results.md                     |  6 +++-
 skills/author-eval-case/SKILL.md              |  9 ++---
 skills/bootstrap-eval-suite/SKILL.md          | 10 +++---
 skills/review-eval-suite/SKILL.md             |  5 +--
 skills/run-eval-suite/SKILL.md                | 11 +++---
 18 files changed, 152 insertions(+), 70 deletions(-)

diff --git a/README.md b/README.md
index c682cca..1229d7c 100644
--- a/README.md
+++ b/README.md
@@ -20,6 +20,22 @@ The package is usable for deterministic local eval suites. Model-assisted comman
 but consumers must configure Promptfoo and calibrate judge behavior before treating model output as
 more than advisory evidence.
 
+## Evaluation policy
+
+Keep evals in three lanes:
+
+- **CI / `pnpm check`:** fast, offline, structural checks only. Suitable checks include
+  format/lint, static docs or schema validation, adapter import/syntax validation, fixture manifest
+  validation, local unit tests for graders/helpers, and seeded fixture checks that never call
+  external providers.
+- **Local on-demand:** deterministic suite runs, including semantic case portfolios, run before
+  significant changes in the consumer repo. These commands may create result bundles and should use
+  the consumer's documented local scripts.
+- **Manual/advisory:** Promptfoo/Codex model-assisted generation, pointwise judging, pairwise
+  judging, LLM judge coverage, long product-to-design or product-to-plan session evals, and
+  expensive replay suites. These require explicit local setup, auth where relevant, and human
+  calibration. They are not default CI gates.
+
 ## What it is
 
 `eval-kit` is a small runner and bootstrap package for repository-local evaluation suites.
@@ -67,23 +83,23 @@ Install from a Git tag in a consumer repo:
 Bootstrap a generic deterministic suite:
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
-eval-kit doctor
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
+pnpm exec eval-kit doctor
 ```
 
 Create a case:
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
-eval-kit list-cases
-eval-kit doctor
+pnpm exec eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit list-cases
+pnpm exec eval-kit doctor
 ```
 
 Run a deterministic case:
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --config evals/eval-kit.config.json \
   --case case-example-v1 \
   --candidate path/to/candidate.md \
@@ -160,7 +176,10 @@ pnpm install --frozen-lockfile
 pnpm check
 ```
 
-`pnpm check` is the required local and CI gate.
+`pnpm check` is the required local and CI gate for fast, offline, structural checks. Keep
+run-producing semantic portfolios, model-assisted generation/judging, LLM judge coverage, pairwise
+judging, and expensive full-case replays out of the default gate unless a consumer repo documents a
+narrow deterministic structural subset.
 
 ## Release model
 
diff --git a/docs/design/bootstrap-cli.md b/docs/design/bootstrap-cli.md
index 623fdea..5e2d5a6 100644
--- a/docs/design/bootstrap-cli.md
+++ b/docs/design/bootstrap-cli.md
@@ -18,8 +18,8 @@ eval-kit list-cases
 Creates a generic deterministic suite skeleton.
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
 ```
 
 Generated files:
@@ -47,7 +47,7 @@ Rules:
 Creates a generic deterministic case.
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit scaffold-case --case case-example-v1
 ```
 
 Generated files:
@@ -83,6 +83,10 @@ Checks:
 - consumer fixture validation hook passes;
 - Promptfoo exists only when model-assisted methods are enabled.
 
+Policy note: `doctor` reports suite validity, not CI policy compliance. Consumer default CI should
+stay fast and offline, without run-producing, model-assisted, networked, or manually calibrated
+evals.
+
 ## `list-cases`
 
 Lists discovered case IDs from the configured case manifests.
diff --git a/docs/design/consumer-integration.md b/docs/design/consumer-integration.md
index e7953f0..990b177 100644
--- a/docs/design/consumer-integration.md
+++ b/docs/design/consumer-integration.md
@@ -27,7 +27,12 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva
 }
 ```
 
-Consumers decide whether evals are part of `pnpm check`. Start with manual scripts until the suite is stable, then add deterministic checks to `pnpm check`.
+Consumers decide which eval checks belong in each lane. Keep `pnpm check` fast, offline, and
+structural: format/lint, static schema/docs validation, adapter import/syntax validation, fixture
+manifest validation, local grader/helper unit tests, and seeded fixture checks that do not call
+external providers. Run deterministic semantic case portfolios locally before significant changes.
+Keep Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long session evals, and
+expensive full-case replays manual/advisory rather than default CI gates.
 
 ## Consumer-owned files
 
@@ -50,9 +55,9 @@ evals/
 - [ ] Bootstrap or create evals/ files.
 - [ ] Replace generic adapter semantics with repo-local semantics.
 - [ ] Add at least one deterministic case.
-- [ ] Run eval-kit doctor.
+- [ ] Run `pnpm exec eval-kit doctor`.
 - [ ] Run one deterministic case.
-- [ ] Decide whether deterministic evals enter pnpm check.
+- [ ] Decide whether any fast, offline structural checks enter `pnpm check`.
 - [ ] Document local eval commands in README or evals/README.md.
 ```
 
@@ -63,3 +68,4 @@ evals/
 - Do not put suite semantics into eval-kit.
 - Do not enable Promptfoo commands without local dependency/auth setup.
 - Do not trust model judges without calibration.
+- Do not require auth, network, external providers, or manual calibration in `pnpm check`.
diff --git a/docs/design/skills.md b/docs/design/skills.md
index 0f27b6a..4f7e2c3 100644
--- a/docs/design/skills.md
+++ b/docs/design/skills.md
@@ -23,10 +23,10 @@ Must guide:
 
 - read repo instructions;
 - add dependency;
-- run `eval-kit init --suite generic --dry-run`;
+- run `pnpm exec eval-kit init --suite generic --dry-run`;
 - inspect planned files;
-- run `eval-kit init --suite generic`;
-- run `eval-kit doctor`;
+- run `pnpm exec eval-kit init --suite generic`;
+- run `pnpm exec eval-kit doctor`;
 - report remaining consumer decisions.
 
 Must not:
@@ -56,6 +56,7 @@ Must detect:
 - hidden requirements in reference outputs;
 - vague rubrics;
 - model judges used as hard gates without calibration;
+- run-producing semantic portfolios treated as default CI gates;
 - cases too large to debug;
 - consumer semantics leaking into eval-kit.
 
diff --git a/docs/design/technical-design-follow-up.md b/docs/design/technical-design-follow-up.md
index ce167d2..556f8f2 100644
--- a/docs/design/technical-design-follow-up.md
+++ b/docs/design/technical-design-follow-up.md
@@ -43,14 +43,14 @@ In `technical-design/evals/adapter.mjs`:
 
 4. Add tests that call these adapter exports and assert required keys exist.
 
-5. Run:
+5. Run the fast/offline gate and local on-demand deterministic smoke:
 
 ```bash
 pnpm check
 pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit
 ```
 
-Optional if local auth exists:
+Manual/advisory if local auth exists:
 
 ```bash
 pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model <model> --provider openai --effort medium --run-id verify-generation-vars
diff --git a/docs/examples.md b/docs/examples.md
index c265225..0def3be 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -21,7 +21,7 @@ evals/
 ## Deterministic Case
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --config evals/eval-kit.config.json \
   --case case-alpha \
   --candidate evals/cases/case-alpha/candidate.md \
@@ -85,7 +85,7 @@ export const renderReport = ({ caseId, grades, findings }) => {
 ## Fixture Validation
 
 ```bash
-eval-kit validate-fixtures --config evals/eval-kit.config.json
+pnpm exec eval-kit validate-fixtures --config evals/eval-kit.config.json
 ```
 
 The kit validates discovered case manifests first. If `adapter.validateFixtures` exists, the kit
@@ -129,9 +129,10 @@ await runCase({
 `generate`, `judge-coverage`, and `judge-pairwise` are optional Promptfoo-backed commands. They
 require a consumer adapter that returns the variables expected by the selected prompt template.
 Consumers can use the bundled generic prompts or set `prompt_templates` to suite-owned prompts.
+Treat these as manual/advisory runs, not default CI gates.
 
 ```bash
-eval-kit judge-coverage \
+pnpm exec eval-kit judge-coverage \
   --config evals/eval-kit.config.json \
   --case case-alpha \
   --candidate evals/cases/case-alpha/candidate.md \
diff --git a/docs/guides/consumer-integration.md b/docs/guides/consumer-integration.md
index cc1bbfc..ce2baaf 100644
--- a/docs/guides/consumer-integration.md
+++ b/docs/guides/consumer-integration.md
@@ -32,9 +32,9 @@ pnpm install
 ## Bootstrap generic files
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
-eval-kit doctor
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
+pnpm exec eval-kit doctor
 ```
 
 ## Replace semantics
@@ -53,7 +53,7 @@ evals/prompts/*            # local prompts, if overriding bundled generic prompt
 
 ## Decide gate level
 
-Start manual:
+Start with package scripts for local use:
 
 ```json
 {
@@ -64,9 +64,19 @@ Start manual:
 }
 ```
 
-Once stable, add deterministic evals to `pnpm check`.
+Use three lanes:
 
-Do not add model-assisted evals to `pnpm check` unless the repo has explicit policy for credentials, cost, determinism, and judge calibration.
+- **CI / `pnpm check`:** fast, offline, structural checks only. This can include format/lint,
+  static docs/schema validation, adapter import or syntax validation, fixture manifest validation,
+  local unit tests for graders/helpers, and seeded fixture checks that do not require auth, network,
+  model calls, or manual calibration.
+- **Local on-demand:** deterministic `run-case` suites and semantic case portfolios. Run these
+  before significant changes, using the consumer repo's scripts or `pnpm exec eval-kit`.
+- **Manual/advisory:** Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long
+  product-to-design/product-to-plan session evals, and expensive full-case replay suites.
+
+Do not add run-producing semantic portfolios or model-assisted evals to the default CI gate. If a
+consumer wants a deterministic subset in `pnpm check`, keep it short, offline, and structural.
 
 ## Review checklist
 
@@ -76,5 +86,5 @@ Do not add model-assisted evals to `pnpm check` unless the repo has explicit pol
 - [ ] Are hidden reference answers avoided?
 - [ ] Are deterministic blockers not averaged away?
 - [ ] Are model judges advisory unless calibrated?
-- [ ] Does pnpm check remain reasonable for contributors?
+- [ ] Does `pnpm check` remain fast, offline, and reasonable for contributors?
 ```
diff --git a/docs/guides/model-assisted-evals.md b/docs/guides/model-assisted-evals.md
index 1a0a22c..e3798db 100644
--- a/docs/guides/model-assisted-evals.md
+++ b/docs/guides/model-assisted-evals.md
@@ -1,6 +1,7 @@
 # Model-assisted evals
 
-Model-assisted evals are optional. Deterministic evals are the default.
+Model-assisted evals are optional manual/advisory checks. Deterministic evals are the default
+local evidence path, and model-assisted commands are never default CI gates.
 
 ## When to use
 
@@ -23,12 +24,12 @@ A consumer repo must provide:
 - adapter hooks for prompt variables;
 - rubrics;
 - result review process;
-- calibration policy before using judge results as gates.
+- calibration policy before using judge results as gates outside default CI.
 
 ## Commands
 
 ```bash
-eval-kit generate \
+pnpm exec eval-kit generate \
   --case <case-id> \
   --model <model> \
   --provider openai \
@@ -36,7 +37,7 @@ eval-kit generate \
   --run-id <run-id>
 
 
-eval-kit judge-coverage \
+pnpm exec eval-kit judge-coverage \
   --case <case-id> \
   --candidate <candidate.md> \
   --model <model> \
@@ -45,7 +46,7 @@ eval-kit judge-coverage \
   --run-id <run-id>
 
 
-eval-kit judge-pairwise \
+pnpm exec eval-kit judge-pairwise \
   --case <case-id> \
   --candidate-a <a.md> \
   --candidate-b <b.md> \
diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md
index cf409cc..7dc40aa 100644
--- a/docs/guides/quickstart.md
+++ b/docs/guides/quickstart.md
@@ -21,22 +21,22 @@ pnpm install
 ## 2. Bootstrap files
 
 ```bash
-eval-kit init --suite generic --dry-run
-eval-kit init --suite generic
+pnpm exec eval-kit init --suite generic --dry-run
+pnpm exec eval-kit init --suite generic
 ```
 
 ## 3. Validate setup
 
 ```bash
-eval-kit doctor
+pnpm exec eval-kit doctor
 ```
 
 ## 4. Create a case
 
 ```bash
-eval-kit scaffold-case --case case-example-v1
-eval-kit list-cases
-eval-kit doctor
+pnpm exec eval-kit scaffold-case --case case-example-v1
+pnpm exec eval-kit list-cases
+pnpm exec eval-kit doctor
 ```
 
 ## 5. Replace placeholders
@@ -57,7 +57,7 @@ The generated adapter checks for exact required text. That is a starter, not a p
 Create or choose a candidate file, then run:
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --case case-example-v1 \
   --candidate path/to/candidate.md \
   --run-id verify-example
@@ -65,6 +65,9 @@ eval-kit run-case \
 
 Results are written under the configured results root.
 
+Deterministic case runs are local on-demand evidence. Consumers should run the documented semantic
+case portfolio before significant changes, but run-producing evals are not default CI gates.
+
 ## 7. Add scripts
 
 Recommended scripts:
@@ -78,3 +81,6 @@ Recommended scripts:
   }
 }
 ```
+
+Package scripts may call the local binary as `eval-kit`. Interactive shell examples use
+`pnpm exec eval-kit` so consumers run the pinned dependency from their repo.
diff --git a/docs/guides/technical-design-integration-fix.md b/docs/guides/technical-design-integration-fix.md
index b9ea0e0..9f979d0 100644
--- a/docs/guides/technical-design-integration-fix.md
+++ b/docs/guides/technical-design-integration-fix.md
@@ -13,7 +13,7 @@ This guide is for the follow-up PR in `agentic-workflow-kit/technical-design`.
 3. Inspect `evals/adapter.mjs`.
 4. Update generation, pointwise, and pairwise variable resolvers to include the generic keys expected by eval-kit prompts.
 5. Add unit tests for adapter variable contracts.
-6. Run the normal gate and deterministic smoke.
+6. Run the fast/offline `pnpm check` gate and the local on-demand deterministic smoke.
 
 ## Required keys
 
@@ -59,7 +59,7 @@ pnpm check
 pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit
 ```
 
-Optional if auth is available:
+Manual/advisory if auth is available:
 
 ```bash
 pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model <model> --provider openai --effort medium --run-id verify-generation-vars
diff --git a/docs/guides/writing-cases.md b/docs/guides/writing-cases.md
index 4f93f89..6c3b85a 100644
--- a/docs/guides/writing-cases.md
+++ b/docs/guides/writing-cases.md
@@ -58,6 +58,12 @@ Expected items should be:
 
 Rubrics are for semantic interpretation. They should not contain hidden hard requirements that are absent from source artifacts.
 
+## Case portfolios
+
+Semantic case portfolios are local on-demand evidence. Keep cases small enough to debug, then run
+the consumer repo's deterministic portfolio before significant changes. Do not make long
+run-producing portfolios, full-case replays, or model-assisted judging part of default CI.
+
 ## Bad signs
 
 Avoid cases where:
diff --git a/docs/reference/cli.md b/docs/reference/cli.md
index bbc0464..6cfa5e6 100644
--- a/docs/reference/cli.md
+++ b/docs/reference/cli.md
@@ -3,15 +3,18 @@
 Run:
 
 ```bash
-eval-kit <command> [options]
+pnpm exec eval-kit <command> [options]
 ```
 
+Package scripts may use plain `eval-kit` because npm/pnpm add local binaries to script `PATH`.
+Interactive consumer shell examples use `pnpm exec eval-kit`.
+
 ## `init`
 
 Create a deterministic generic eval skeleton.
 
 ```bash
-eval-kit init --suite generic [--dry-run] [--force]
+pnpm exec eval-kit init --suite generic [--dry-run] [--force]
 ```
 
 Options:
@@ -27,7 +30,7 @@ Options:
 Create a generic deterministic case skeleton.
 
 ```bash
-eval-kit scaffold-case --case <id> [--config <path>] [--dry-run] [--force]
+pnpm exec eval-kit scaffold-case --case <id> [--config <path>] [--dry-run] [--force]
 ```
 
 Options:
@@ -44,7 +47,7 @@ Options:
 Validate suite setup.
 
 ```bash
-eval-kit doctor [--config <path>]
+pnpm exec eval-kit doctor [--config <path>]
 ```
 
 ## `list-cases`
@@ -52,7 +55,7 @@ eval-kit doctor [--config <path>]
 List discovered case IDs.
 
 ```bash
-eval-kit list-cases [--config <path>]
+pnpm exec eval-kit list-cases [--config <path>]
 ```
 
 ## `run-case`
@@ -60,7 +63,7 @@ eval-kit list-cases [--config <path>]
 Run a deterministic case.
 
 ```bash
-eval-kit run-case \
+pnpm exec eval-kit run-case \
   --case <id> \
   --candidate <path> \
   [--run-id <id>] \
@@ -74,7 +77,7 @@ Exits non-zero when the deterministic verdict is `red`.
 Validate case manifests and call the consumer `validateFixtures` hook when present.
 
 ```bash
-eval-kit validate-fixtures [--config <path>]
+pnpm exec eval-kit validate-fixtures [--config <path>]
 ```
 
 ## `generate`
@@ -82,7 +85,7 @@ eval-kit validate-fixtures [--config <path>]
 Run Promptfoo-backed candidate generation.
 
 ```bash
-eval-kit generate \
+pnpm exec eval-kit generate \
   --case <id> \
   --model <name> \
   --provider <openai|openai:codex-app-server> \
@@ -91,14 +94,14 @@ eval-kit generate \
   [--config <path>]
 ```
 
-Requires Promptfoo and local Codex auth.
+Requires Promptfoo and local Codex auth. This is manual/advisory evidence, not a default CI gate.
 
 ## `judge-coverage`
 
 Run Promptfoo-backed pointwise coverage judging.
 
 ```bash
-eval-kit judge-coverage \
+pnpm exec eval-kit judge-coverage \
   --case <id> \
   --candidate <path> \
   --model <name> \
@@ -118,7 +121,7 @@ calling the consumer adapter. The adapter receives displayed Candidate A/B conte
 own second randomization pass.
 
 ```bash
-eval-kit judge-pairwise \
+pnpm exec eval-kit judge-pairwise \
   --case <id> \
   --candidate-a <path> \
   --candidate-b <path> \
@@ -137,7 +140,7 @@ Fails closed when `methods.judge_pairwise.enabled` is explicitly `false`.
 Compile existing run bundles into a unified report through the consumer hook.
 
 ```bash
-eval-kit report \
+pnpm exec eval-kit report \
   --run-id <id> \
   [--generate <id>] \
   [--deterministic <id>] \
diff --git a/docs/reference/config.md b/docs/reference/config.md
index 230bdc3..ae852d5 100644
--- a/docs/reference/config.md
+++ b/docs/reference/config.md
@@ -58,6 +58,20 @@ evals/eval-kit.config.json
 | `methods`          | Enabled commands and method-specific settings.                |
 | `prompt_templates` | Optional overrides for bundled Promptfoo prompts.             |
 
+## Method categories
+
+Method settings do not decide CI policy by themselves. Consumers should map commands into these
+lanes:
+
+- `doctor`, `list-cases`, `validate-fixtures`, adapter import/syntax checks, static schema/docs
+  validation, and local grader/helper unit tests are suitable for `pnpm check` when they stay fast
+  and offline.
+- `run-case` is deterministic local on-demand evidence for semantic case portfolios. Run it before
+  significant changes, but do not make long run-producing portfolios a default CI gate.
+- `generate`, `judge_coverage`, and `judge_pairwise` are manual/advisory model-assisted methods.
+  They require explicit local setup and must not require auth, network, Promptfoo provider calls,
+  Codex/OpenAI calls, LLM judging, or manual calibration in `pnpm check`.
+
 ## Path rules
 
 - Config path is resolved from the current working directory.
diff --git a/docs/reference/results.md b/docs/reference/results.md
index d90576c..331202c 100644
--- a/docs/reference/results.md
+++ b/docs/reference/results.md
@@ -10,6 +10,10 @@ These commands create run directories:
 - `judge-pairwise`
 - `report`
 
+Run-producing commands are local on-demand or manual/advisory evidence. They are not default CI
+gates. Keep `pnpm check` limited to fast, offline, structural validation unless a consumer
+documents a narrow deterministic subset that does not call external providers.
+
 Setup and inspection commands do not create result bundles. `init` and `scaffold-case` write suite
 or case files, while `doctor`, `list-cases`, and `validate-fixtures` validate or report current
 state without writing `<results_root>/<run-id>/manifest.json`.
@@ -42,7 +46,7 @@ Current schema:
   "git": {
     "commit": "abc123"
   },
-  "command": "eval-kit run-case ...",
+  "command": "pnpm exec eval-kit run-case ...",
   "tool_versions": {
     "node": "v26.0.0",
     "pnpm": "11.9.0"
diff --git a/skills/author-eval-case/SKILL.md b/skills/author-eval-case/SKILL.md
index 2c6af30..931b3e7 100644
--- a/skills/author-eval-case/SKILL.md
+++ b/skills/author-eval-case/SKILL.md
@@ -10,17 +10,18 @@ Use this skill when creating or revising a case in a repo that already has an ev
 ## Steps
 
 1. Read the consumer repo instructions and the local eval-kit config.
-2. Choose a safe case id and run `eval-kit scaffold-case --case <id>`.
+2. Choose a safe case id and run `pnpm exec eval-kit scaffold-case --case <id>`.
 3. Fill the generated `input.md`, `expected-items.json`, and `rubric.md` with consumer-owned content.
-4. Run `eval-kit list-cases` to confirm discovery.
-5. Run `eval-kit doctor` to validate manifests and artifacts.
-6. If a deterministic candidate is available, run `eval-kit run-case --case <id> --candidate <path>`.
+4. Run `pnpm exec eval-kit list-cases` to confirm discovery.
+5. Run `pnpm exec eval-kit doctor` to validate manifests and artifacts.
+6. If a deterministic candidate is available, run `pnpm exec eval-kit run-case --case <id> --candidate <path>`.
 
 ## Boundaries
 
 - Keep domain facts, rubrics, expected items, and pass/fail policy in the consumer repo.
 - Do not move consumer semantics into `@agentic-workflow-kit/eval-kit`.
 - Avoid model-assisted judging until the consumer has calibrated prompts, rubrics, and acceptance policy.
+- Keep semantic portfolio runs local on-demand before significant changes, not default CI.
 - Search for sibling cases before introducing a new fixture shape.
 
 ## Evidence
diff --git a/skills/bootstrap-eval-suite/SKILL.md b/skills/bootstrap-eval-suite/SKILL.md
index bf750fe..7c60391 100644
--- a/skills/bootstrap-eval-suite/SKILL.md
+++ b/skills/bootstrap-eval-suite/SKILL.md
@@ -11,9 +11,9 @@ Use this skill when adding `@agentic-workflow-kit/eval-kit` harness files to a c
 
 1. Read the consumer repo instructions and identify the repo root before writing.
 2. Confirm the consumer actually wants eval-kit; do not add it because a template exists.
-3. Run `eval-kit init --suite generic --dry-run` and inspect the planned files.
-4. Run `eval-kit init --suite generic` only when the dry-run matches the intended scope.
-5. Run `eval-kit doctor` and record the command result.
+3. Run `pnpm exec eval-kit init --suite generic --dry-run` and inspect the planned files.
+4. Run `pnpm exec eval-kit init --suite generic` only when the dry-run matches the intended scope.
+5. Run `pnpm exec eval-kit doctor` and record the command result.
 
 ## Boundaries
 
@@ -21,8 +21,10 @@ Use this skill when adding `@agentic-workflow-kit/eval-kit` harness files to a c
 - The consumer owns cases, prompts, rubrics, graders, pass/fail policy, and product vocabulary.
 - Do not add technical-design, define-product, design-to-plan, jig, or learning-loop semantics by default.
 - Do not install Promptfoo or require Codex auth for deterministic bootstrap.
+- Do not add run-producing, model-assisted, networked, or manually calibrated evals to default CI.
 - Use `--force` only after inspecting existing files and deciding the overwrite is intended.
 
 ## Evidence
 
-Report the repo path, generated files, `eval-kit doctor` result, and any consumer decisions still needed.
+Report the repo path, generated files, `pnpm exec eval-kit doctor` result, and any consumer
+decisions still needed.
diff --git a/skills/review-eval-suite/SKILL.md b/skills/review-eval-suite/SKILL.md
index 8ade167..b1f5135 100644
--- a/skills/review-eval-suite/SKILL.md
+++ b/skills/review-eval-suite/SKILL.md
@@ -10,8 +10,8 @@ Use this skill when auditing or reviewing an eval-kit suite.
 ## Steps
 
 1. Read the consumer repo instructions, eval-kit config, and relevant product or eval docs.
-2. Run `eval-kit doctor` and treat failures as primary evidence.
-3. Run `eval-kit list-cases` and inspect representative case manifests and artifacts.
+2. Run `pnpm exec eval-kit doctor` and treat failures as primary evidence.
+3. Run `pnpm exec eval-kit list-cases` and inspect representative case manifests and artifacts.
 4. Check whether generated or shared files contain consumer semantics that should stay local, or local files duplicate kit mechanics unnecessarily.
 5. If reviewing a change, verify overwrite behavior and generated files with a dry run where relevant.
 
@@ -20,6 +20,7 @@ Use this skill when auditing or reviewing an eval-kit suite.
 - Mechanics belong in eval-kit; product vocabulary, rubric policy, cases, and expected outputs belong in the consumer.
 - Do not require Promptfoo for deterministic suites.
 - Treat model-assisted judging as advisory unless the consumer documents calibration and acceptance thresholds.
+- Treat run-producing semantic portfolios as local on-demand evidence before significant changes, not default CI.
 - Do not claim suite readiness without command evidence.
 
 ## Evidence
diff --git a/skills/run-eval-suite/SKILL.md b/skills/run-eval-suite/SKILL.md
index a4068e8..9ac13f2 100644
--- a/skills/run-eval-suite/SKILL.md
+++ b/skills/run-eval-suite/SKILL.md
@@ -10,17 +10,20 @@ Use this skill when executing a local eval-kit suite.
 ## Steps
 
 1. Read the consumer repo instructions and eval-kit config.
-2. Run `eval-kit doctor` before suite execution.
-3. Run `eval-kit list-cases` and select the requested case ids or the repo-documented subset.
-4. For deterministic runs, execute `eval-kit run-case --case <id> --candidate <path>` for each selected case.
-5. Compile reports with `eval-kit report` when the consumer workflow asks for a combined summary.
+2. Run `pnpm exec eval-kit doctor` before suite execution.
+3. Run `pnpm exec eval-kit list-cases` and select the requested case ids or the repo-documented subset.
+4. For deterministic runs, execute `pnpm exec eval-kit run-case --case <id> --candidate <path>` for each selected case.
+5. Compile reports with `pnpm exec eval-kit report` when the consumer workflow asks for a combined summary.
 6. Run model-assisted commands only when explicitly requested and configured.
 
 ## Boundaries
 
 - Deterministic evidence can support pass/fail claims according to the consumer policy.
+- Semantic case portfolios are local on-demand checks before significant changes, not default CI.
 - Model-assisted generation, coverage judging, and pairwise judging are advisory unless calibrated.
 - Do not require Codex auth or Promptfoo for deterministic-only runs.
+- Do not put Promptfoo provider calls, Codex/OpenAI calls, LLM judging, pairwise judging, or long
+  replay suites in `pnpm check`.
 - Preserve raw outputs according to the consumer repo's artifact policy.
 
 ## Evidence