agentic-workflow-kit · aryeko · Jul 3, 2026 · Jul 3, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,10 +9,24 @@ documented with migration notes.
 
 ### Planned
 
-- Consumer integration hardening.
 - Additional docs for suite-specific adoption.
 - Better compatibility tests for Promptfoo variable contracts.
 
+## [0.1.3] - 2026-07-04
+
+### Added
+
+- Documented the standard two-config pattern for deterministic and manual pointwise model-judge
+  consumer eval lanes.
+- Updated eval-kit skills to direct agents toward manual model-judge configs while keeping provider
+  calls out of default CI gates.
+
+### Notes
+
+- No runtime, schema, or CLI behavior changed.
+- No npm package is published.
+- Consumers may pin `github:agentic-workflow-kit/eval-kit#v0.1.3`.
+
 ## [0.1.2] - 2026-07-03
 
 ### Fixed
@@ -63,7 +77,8 @@ documented with migration notes.
 - Suite-specific presets remain deferred.
 - Consumer repos own their own semantics, prompts, cases, and pass/fail policies.
 
-[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...main
+[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.3...main
+[0.1.3]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/agentic-workflow-kit/eval-kit/releases/tag/v0.1.0
diff --git a/README.md b/README.md
@@ -11,7 +11,7 @@ Shared evaluation infrastructure for `agentic-workflow-kit` repositories.
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
   }
 }
 ```
@@ -70,7 +70,7 @@ Install from a Git tag in a consumer repo:
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
   },
   "scripts": {
     "eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json",
@@ -155,6 +155,7 @@ Start here:
 - [`docs/design/README.md`](docs/design/README.md) - architecture, decisions, and contracts.
 - [`docs/guides/quickstart.md`](docs/guides/quickstart.md) - bootstrap and first deterministic run.
 - [`docs/guides/consumer-integration.md`](docs/guides/consumer-integration.md) - add eval-kit to another repo.
+- [`docs/guides/model-assisted-evals.md`](docs/guides/model-assisted-evals.md) - configure manual model-judge lanes.
 - [`docs/reference/adapter-contract.md`](docs/reference/adapter-contract.md) - adapter exports and hook shapes.
 - [`docs/reference/release-process.md`](docs/reference/release-process.md) - version and tag process.
 
@@ -189,6 +190,7 @@ For now, releases are Git tags that consumers pin in `package.json`:
 v0.1.0
 v0.1.1
 v0.1.2
+v0.1.3
 v0.2.0
 ```
 

diff --git a/docs/design/consumer-integration.md b/docs/design/consumer-integration.md
@@ -9,7 +9,7 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
   }
 }
 ```
@@ -34,6 +34,13 @@ external providers. Run deterministic semantic case portfolios locally before si
 Keep Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long session evals, and
 expensive full-case replays manual/advisory rather than default CI gates.
 
+Pointwise model-judge consumers should use a separate manual config instead of toggling model
+methods in the deterministic default config. Keep `evals/eval-kit.config.json` deterministic and
+CI-safe. Put `judge_coverage.enabled=true` in `evals/eval-kit.model-judge.config.json`, keep
+`generate`, `judge_pairwise`, and `report` disabled there, and route `eval:judge:*` scripts through
+that manual config. Add pairwise later only when the consumer has calibrated pointwise evidence and a
+specific comparison need.
+
 ## Consumer-owned files
 
 ```text

diff --git a/docs/guides/consumer-integration.md b/docs/guides/consumer-integration.md
@@ -18,7 +18,7 @@ If you cannot state the eval goal, do not bootstrap a suite yet. Empty harnesses
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
   }
 }
 ```
@@ -78,6 +78,26 @@ Use three lanes:
 Do not add run-producing semantic portfolios or model-assisted evals to the default CI gate. If a
 consumer wants a deterministic subset in `pnpm check`, keep it short, offline, and structural.
 
+## Add manual pointwise judging
+
+When a consumer is ready for pointwise model judging, add a separate
+`evals/eval-kit.model-judge.config.json` instead of toggling model methods in the default
+deterministic config.
+
+Standard scripts:
+
+```json
+{
+  "eval:judge:doctor": "eval-kit doctor --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:list": "eval-kit list-cases --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:validate-fixtures": "eval-kit validate-fixtures --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:coverage": "eval-kit judge-coverage --config evals/eval-kit.model-judge.config.json"
+}
+```
+
+Keep `generate`, `judge_pairwise`, and `report` disabled in this manual config. Add pairwise later
+with a separate config only after the consumer has calibrated pointwise evidence.
+
 ## Review checklist
 
 ```text

diff --git a/docs/guides/model-assisted-evals.md b/docs/guides/model-assisted-evals.md
@@ -26,6 +26,33 @@ A consumer repo must provide:
 - result review process;
 - calibration policy before using judge results as gates outside default CI.
 
+## Standard consumer config
+
+Use two configs for pointwise model judging:
+
+- `evals/eval-kit.config.json` is the default deterministic config. It is the only config that
+  belongs in `pnpm check` or CI. Keep `generate`, `judge_coverage`, and `judge_pairwise` disabled.
+- `evals/eval-kit.model-judge.config.json` is the manual pointwise config. Enable
+  `judge_coverage`, disable `generate`, `judge_pairwise`, and `report`, and include only the cases
+  the consumer has chosen for manual judge calibration.
+
+Use stable script names so agents can run the right config without guessing:
+
+```json
+{
+  "eval:judge:doctor": "eval-kit doctor --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:list": "eval-kit list-cases --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:validate-fixtures": "eval-kit validate-fixtures --config evals/eval-kit.model-judge.config.json",
+  "eval:judge:coverage": "eval-kit judge-coverage --config evals/eval-kit.model-judge.config.json"
+}
+```
+
+Do not ask contributors to temporarily flip `judge_coverage` in the deterministic config. That
+creates drift between local manual runs and CI-safe defaults.
+
+Pairwise judging is a separate follow-up lane. Add `evals/eval-kit.pairwise.config.json` only after
+the consumer has calibrated pointwise evidence and a specific comparison need.
+
 ## Commands
 
 ```bash
@@ -38,6 +65,7 @@ pnpm exec eval-kit generate \
 
 
 pnpm exec eval-kit judge-coverage \
+  --config evals/eval-kit.model-judge.config.json \
   --case <case-id> \
   --candidate <candidate.md> \
   --model <model> \

diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md
@@ -7,7 +7,7 @@ This guide adds a generic deterministic eval suite to a consumer repo.
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
   }
 }
 ```

diff --git a/docs/reference/release-process.md b/docs/reference/release-process.md
@@ -34,7 +34,7 @@ Consumers depend on tags like:
 Title:
 
 ```text
-chore(release): v0.1.2
+chore(release): v0.1.3
 ```
 
 Required changes:
@@ -63,18 +63,18 @@ git checkout main
 git pull --ff-only
 git rev-parse HEAD
 
-git tag -a v0.1.2 -m "v0.1.2"
-git push origin v0.1.2
+git tag -a v0.1.3 -m "v0.1.3"
+git push origin v0.1.3
 ```
 
 Verify:
 
 ```bash
-git rev-parse v0.1.2^{}
-git show --no-patch --decorate v0.1.2
+git rev-parse v0.1.3^{}
+git show --no-patch --decorate v0.1.3
 ```
 
-`v0.1.2^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.2`
+`v0.1.3^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.3`
 returns the tag object; `^{}` dereferences to the commit.
 
 ## GitHub Release
@@ -93,7 +93,7 @@ For each consumer repo:
 
 ```json
 {
-  "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+  "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
 }
 ```
 
@@ -108,7 +108,7 @@ pnpm check
 3. Run consumer smoke commands, for example in `technical-design`:
 
 ```bash
-pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.2
+pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.3
 ```
 
 4. Open a PR with dependency, lockfile, and any compatibility fixes.
@@ -120,7 +120,7 @@ Do not move the tag.
 Create a new patch release:
 
 ```text
-v0.1.2 -> v0.1.3
+v0.1.3 -> v0.1.4
 ```
 
 Then open consumer bump PRs.

diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@agentic-workflow-kit/eval-kit",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "description": "Portable eval runner primitives for local eval suites.",
   "private": true,
   "type": "module",

diff --git a/skills/bootstrap-eval-suite/SKILL.md b/skills/bootstrap-eval-suite/SKILL.md
@@ -15,13 +15,26 @@ Use this skill when adding `@agentic-workflow-kit/eval-kit` harness files to a c
 4. Run `pnpm exec eval-kit init --suite generic` only when the dry-run matches the intended scope.
 5. Run `pnpm exec eval-kit doctor` and record the command result.
 
+## Manual Model-Judge Config
+
+When adding pointwise model judging to a consumer that already has deterministic evals, use the
+standard two-config pattern:
+
+- Keep `evals/eval-kit.config.json` deterministic and CI-safe, with model-assisted methods disabled.
+- Add `evals/eval-kit.model-judge.config.json` for manual pointwise judging, with
+  `judge_coverage.enabled=true` and `generate`, `judge_pairwise`, and `report` disabled.
+- Add scripts named `eval:judge:doctor`, `eval:judge:list`, `eval:judge:validate-fixtures`, and
+  `eval:judge:coverage` that point at the manual config.
+- Keep pairwise out of this standard setup. Add a separate pairwise config only after calibration.
+
 ## Boundaries
 
 - The kit owns loading, validation, discovery, result plumbing, and generic bootstrap mechanics.
 - The consumer owns cases, prompts, rubrics, graders, pass/fail policy, and product vocabulary.
 - Do not add technical-design, define-product, design-to-plan, jig, or learning-loop semantics by default.
 - Do not install Promptfoo or require Codex auth for deterministic bootstrap.
 - Do not add run-producing, model-assisted, networked, or manually calibrated evals to default CI.
+- Do not tell users to edit the deterministic config to enable manual judge runs.
 - Use `--force` only after inspecting existing files and deciding the overwrite is intended.
 
 ## Evidence

diff --git a/skills/review-eval-suite/SKILL.md b/skills/review-eval-suite/SKILL.md
@@ -13,13 +13,17 @@ Use this skill when auditing or reviewing an eval-kit suite.
 2. Run `pnpm exec eval-kit doctor` and treat failures as primary evidence.
 3. Run `pnpm exec eval-kit list-cases` and inspect representative case manifests and artifacts.
 4. Check whether generated or shared files contain consumer semantics that should stay local, or local files duplicate kit mechanics unnecessarily.
-5. If reviewing a change, verify overwrite behavior and generated files with a dry run where relevant.
+5. If model judging is present, confirm the default config stays deterministic and a separate manual
+   model-judge config owns pointwise `judge_coverage`.
+6. If reviewing a change, verify overwrite behavior and generated files with a dry run where relevant.
 
 ## Boundaries
 
 - Mechanics belong in eval-kit; product vocabulary, rubric policy, cases, and expected outputs belong in the consumer.
 - Do not require Promptfoo for deterministic suites.
 - Treat model-assisted judging as advisory unless the consumer documents calibration and acceptance thresholds.
+- Prefer `evals/eval-kit.model-judge.config.json` for manual pointwise judging; do not require users
+  to flip enabled flags in the deterministic config.
 - Treat run-producing semantic portfolios as local on-demand evidence before significant changes, not default CI.
 - Do not claim suite readiness without command evidence.
 

diff --git a/skills/run-eval-suite/SKILL.md b/skills/run-eval-suite/SKILL.md
@@ -14,7 +14,8 @@ Use this skill when executing a local eval-kit suite.
 3. Run `pnpm exec eval-kit list-cases` and select the requested case ids or the repo-documented subset.
 4. For deterministic runs, execute `pnpm exec eval-kit run-case --case <id> --candidate <path>` for each selected case.
 5. Compile reports with `pnpm exec eval-kit report` when the consumer workflow asks for a combined summary.
-6. Run model-assisted commands only when explicitly requested and configured.
+6. Run pointwise model-assisted commands only when explicitly requested, configured, and routed
+   through `evals/eval-kit.model-judge.config.json`.
 
 ## Boundaries
 
@@ -24,8 +25,11 @@ Use this skill when executing a local eval-kit suite.
 - Do not require Codex auth or Promptfoo for deterministic-only runs.
 - Do not put Promptfoo provider calls, Codex/OpenAI calls, LLM judging, pairwise judging, or long
   replay suites in `pnpm check`.
+- Use the consumer's `eval:judge:doctor`, `eval:judge:list`, and `eval:judge:validate-fixtures`
+  scripts before any manual `eval:judge:coverage` run.
 - Preserve raw outputs according to the consumer repo's artifact policy.
 
 ## Evidence
 
-Report the config path, cases run, result directories, verdicts, report paths, and any skipped or advisory-only checks.
+Report the config path, cases run, result directories, verdicts, report paths, and any skipped or
+advisory-only checks. For model-assisted runs, state that provider calls were explicitly requested.
diff --git a/src/sdk.mjs b/src/sdk.mjs
@@ -15,7 +15,7 @@ import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs";
 const DEFAULT_SANDBOX_MODE = "read-only";
 const DEFAULT_APPROVAL_POLICY = "never";
 const RANDOMIZATION_METHOD = "sha256-seed-parity-v1";
-const EVAL_KIT_VERSION = "0.1.2";
+const EVAL_KIT_VERSION = "0.1.3";
 
 const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
-Original file line number
+Diff line change
@@ Expand Up @@
     ```json
     {
       "devDependencies": {
-        "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2"
+        "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3"
       }
     }
     ```
@@ Expand Down @@