chore: bump technical-design eval-kit to v0.1.1

docs/design/promptfoo-codex-eval-pilot.md

@@ -146,29 +146,37 @@ product is designed and governed. Executable eval assets, fixture data, generate
 and eval-specific implementation docs belong in `evals/`.
 
 The root package stays small. It keeps `check` as the public deterministic repo gate and exposes
-manual eval commands that call the internal kit:
+only the basic deterministic eval-kit commands. Model-assisted commands remain direct `eval-kit`
+CLI usage because they require local auth, explicit calibration, and human interpretation:
 
 ```text
+pnpm eval:doctor
+pnpm eval:list
 pnpm eval:case -- <args>
-pnpm eval:generate -- <args>
-pnpm eval:judge -- <args>
-pnpm eval:judge:coverage -- <args>
-pnpm eval:manual-report -- <args>
+pnpm exec eval-kit generate --config evals/eval-kit.config.json <args>
+pnpm exec eval-kit judge-coverage --config evals/eval-kit.config.json <args>
+pnpm exec eval-kit judge-pairwise --config evals/eval-kit.config.json <args>
+pnpm exec eval-kit report --config evals/eval-kit.config.json <args>
 ```
 
 The root `pnpm check` command remains the public repository gate. It should run formatting and the
 eval package's deterministic check.
 
 ## Planned Suite Shape
 
-Add `promptfoo` as a development dependency of the eval package and add manual scripts:
-
-- `eval:generate` - run the candidate-generation Promptfoo suite.
-- `eval:judge:coverage` - run the pointwise coverage judge Promptfoo suite.
-- `eval:judge` - run the pairwise judge Promptfoo suite.
-- `eval:outcome` - validate and report a redacted outcome-study summary.
-- `eval:manual-report` - combine generation, deterministic, judge, and outcome results into one
-  final report.
+Keep Promptfoo installed for optional direct CLI runs, but do not add package scripts for
+model-assisted commands until there is a calibrated workflow. Because model-assisted methods are
+disabled in `evals/eval-kit.config.json` by default, a manual run must intentionally enable the
+specific method before invoking it:
+
+- `pnpm exec eval-kit generate` - run the candidate-generation Promptfoo suite after enabling
+  `methods.generate`.
+- `pnpm exec eval-kit judge-coverage` - run the pointwise coverage judge Promptfoo suite after
+  enabling `methods.judge_coverage`.
+- `pnpm exec eval-kit judge-pairwise` - run the pairwise judge Promptfoo suite after enabling
+  `methods.judge_pairwise`.
+- `pnpm exec eval-kit report` - combine generation, deterministic, judge, and outcome results into
+  one final report.
 
 Keep `pnpm check` deterministic-only.
 

evals/README.md

@@ -101,10 +101,15 @@ Check local auth:
 codex login status
 ```
 
+Before running generation or pointwise coverage judging, set `methods.generate.enabled` and
+`methods.judge_coverage.enabled` to `true` in `evals/eval-kit.config.json`. Pairwise judging also
+requires `methods.judge_pairwise.enabled` to be set to `true`. Keep these methods disabled for the
+default deterministic contributor gate.
+
 Generate a candidate design:
 
 ```bash
-pnpm eval:generate -- --case <case-id> --model gpt-5.4 --provider openai --effort medium --run-id <run-id>-generate
+pnpm exec eval-kit generate --config evals/eval-kit.config.json --case <case-id> --model gpt-5.4 --provider openai --effort medium --run-id <run-id>-generate
 ```
 
 Grade deterministically:
@@ -116,20 +121,19 @@ pnpm eval:case -- --case <case-id> --candidate evals/results/<run-id>-generate/c
 Run pointwise coverage judging:
 
 ```bash
-pnpm eval:judge:coverage -- --case <case-id> --candidate evals/results/<run-id>-generate/cases/<case-id>/candidate.md --model gpt-5.4 --provider openai --effort medium --run-id <run-id>-pointwise
+pnpm exec eval-kit judge-coverage --config evals/eval-kit.config.json --case <case-id> --candidate evals/results/<run-id>-generate/cases/<case-id>/candidate.md --model gpt-5.4 --provider openai --effort medium --run-id <run-id>-pointwise
 ```
 
-Run pairwise judging only after deterministic blockers are understood and
-`methods.judge_pairwise.enabled` is set to `true` in `eval-kit.config.json`:
+Run pairwise judging only after deterministic blockers are understood:
 
 ```bash
-pnpm eval:judge -- --case <case-id> --candidate-a <a.md> --candidate-b <b.md> --model gpt-5.4 --provider openai --effort medium --seed <seed> --run-id <run-id>-pairwise
+pnpm exec eval-kit judge-pairwise --config evals/eval-kit.config.json --case <case-id> --candidate-a <a.md> --candidate-b <b.md> --model gpt-5.4 --provider openai --effort medium --seed <seed> --run-id <run-id>-pairwise
 ```
 
 Write a combined manual report:
 
 ```bash
-pnpm eval:manual-report -- --run-id <run-id> --generate <run-id>-generate --deterministic <run-id>-deterministic --judge-coverage <run-id>-pointwise
+pnpm exec eval-kit report --config evals/eval-kit.config.json --run-id <run-id> --generate <run-id>-generate --deterministic <run-id>-deterministic --judge-coverage <run-id>-pointwise
 ```
 
 Generated outputs go under `evals/results/<run-id>/` and are ignored by default. A result bundle

evals/eval-kit.config.json

@@ -16,13 +16,10 @@
       "reporter": "markdown"
     },
     "generate": {
-      "enabled": true,
-      "prompt": "@eval-kit/generation"
+      "enabled": false
     },
     "judge_coverage": {
-      "enabled": true,
-      "prompt": "@eval-kit/pointwise",
-      "rubric": "case:rubric.md"
+      "enabled": false
     },
     "judge_pairwise": {
       "enabled": false

evals/results/README.md

@@ -1,6 +1,6 @@
 # Eval Results
 
-Generated eval outputs belong here under `packages/evals/results/<run-id>/`.
+Generated eval outputs belong here under `evals/results/<run-id>/`.
 
 This directory is intentionally ignored except for this README. Commit source fixtures, schemas, and
 small expected outputs; keep raw run outputs local unless a human reviewer explicitly promotes a
@@ -9,7 +9,7 @@ redacted summary.
 Expected layout:
 
 ```text
-packages/evals/results/<run-id>/
+evals/results/<run-id>/
   manifest.json
   report.md
   grades.json                    # deterministic case runs only

package.json

@@ -36,16 +36,14 @@
     "check": "pnpm lint && pnpm check:non-eval && pnpm smoke:enforce-generator && pnpm check:deterministic",
     "check:deterministic": "pnpm check:eval-static && pnpm eval:unit && pnpm eval:enforce",
     "check:eval-static": "node scripts/check_eval_static.mjs",
+    "eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json",
+    "eval:list": "eval-kit list-cases --config evals/eval-kit.config.json",
     "eval:unit": "vitest run evals/tests",
     "eval:case": "eval-kit run-case --config evals/eval-kit.config.json",
-    "eval:enforce": "node scripts/run_enforce_eval.mjs",
-    "eval:generate": "eval-kit generate --config evals/eval-kit.config.json",
-    "eval:judge": "eval-kit judge-pairwise --config evals/eval-kit.config.json",
-    "eval:judge:coverage": "eval-kit judge-coverage --config evals/eval-kit.config.json",
-    "eval:manual-report": "eval-kit report --config evals/eval-kit.config.json"
+    "eval:enforce": "node scripts/run_enforce_eval.mjs"
   },
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.0",
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.1",
     "ajv": "^8.17.1",
     "dependency-cruiser": "^18.0.0",
     "prettier": "^3.6.2",