diff --git a/CHANGELOG.md b/CHANGELOG.md index 77b75e6..944000f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,10 +9,24 @@ documented with migration notes. ### Planned -- Consumer integration hardening. - Additional docs for suite-specific adoption. - Better compatibility tests for Promptfoo variable contracts. +## [0.1.3] - 2026-07-04 + +### Added + +- Documented the standard two-config pattern for deterministic and manual pointwise model-judge + consumer eval lanes. +- Updated eval-kit skills to direct agents toward manual model-judge configs while keeping provider + calls out of default CI gates. + +### Notes + +- No runtime, schema, or CLI behavior changed. +- No npm package is published. +- Consumers may pin `github:agentic-workflow-kit/eval-kit#v0.1.3`. + ## [0.1.2] - 2026-07-03 ### Fixed @@ -63,7 +77,8 @@ documented with migration notes. - Suite-specific presets remain deferred. - Consumer repos own their own semantics, prompts, cases, and pass/fail policies. -[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...main +[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.3...main +[0.1.3]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...v0.1.3 [0.1.2]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.1...v0.1.2 [0.1.1]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.0...v0.1.1 [0.1.0]: https://github.com/agentic-workflow-kit/eval-kit/releases/tag/v0.1.0 diff --git a/README.md b/README.md index b31729e..9cce4e3 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Shared evaluation infrastructure for `agentic-workflow-kit` repositories. ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" } } ``` @@ -70,7 +70,7 @@ Install from a Git tag in a consumer repo: ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" }, "scripts": { "eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json", @@ -155,6 +155,7 @@ Start here: - [`docs/design/README.md`](docs/design/README.md) - architecture, decisions, and contracts. - [`docs/guides/quickstart.md`](docs/guides/quickstart.md) - bootstrap and first deterministic run. - [`docs/guides/consumer-integration.md`](docs/guides/consumer-integration.md) - add eval-kit to another repo. +- [`docs/guides/model-assisted-evals.md`](docs/guides/model-assisted-evals.md) - configure manual model-judge lanes. - [`docs/reference/adapter-contract.md`](docs/reference/adapter-contract.md) - adapter exports and hook shapes. - [`docs/reference/release-process.md`](docs/reference/release-process.md) - version and tag process. @@ -189,6 +190,7 @@ For now, releases are Git tags that consumers pin in `package.json`: v0.1.0 v0.1.1 v0.1.2 +v0.1.3 v0.2.0 ``` diff --git a/docs/design/consumer-integration.md b/docs/design/consumer-integration.md index cb5d8b9..cae5127 100644 --- a/docs/design/consumer-integration.md +++ b/docs/design/consumer-integration.md @@ -9,7 +9,7 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" } } ``` @@ -34,6 +34,13 @@ external providers. Run deterministic semantic case portfolios locally before si Keep Promptfoo/Codex generation, LLM judge coverage, pairwise judging, long session evals, and expensive full-case replays manual/advisory rather than default CI gates. +Pointwise model-judge consumers should use a separate manual config instead of toggling model +methods in the deterministic default config. Keep `evals/eval-kit.config.json` deterministic and +CI-safe. Put `judge_coverage.enabled=true` in `evals/eval-kit.model-judge.config.json`, keep +`generate`, `judge_pairwise`, and `report` disabled there, and route `eval:judge:*` scripts through +that manual config. Add pairwise later only when the consumer has calibrated pointwise evidence and a +specific comparison need. + ## Consumer-owned files ```text diff --git a/docs/guides/consumer-integration.md b/docs/guides/consumer-integration.md index be71845..7cdd389 100644 --- a/docs/guides/consumer-integration.md +++ b/docs/guides/consumer-integration.md @@ -18,7 +18,7 @@ If you cannot state the eval goal, do not bootstrap a suite yet. Empty harnesses ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" } } ``` @@ -78,6 +78,26 @@ Use three lanes: Do not add run-producing semantic portfolios or model-assisted evals to the default CI gate. If a consumer wants a deterministic subset in `pnpm check`, keep it short, offline, and structural. +## Add manual pointwise judging + +When a consumer is ready for pointwise model judging, add a separate +`evals/eval-kit.model-judge.config.json` instead of toggling model methods in the default +deterministic config. + +Standard scripts: + +```json +{ + "eval:judge:doctor": "eval-kit doctor --config evals/eval-kit.model-judge.config.json", + "eval:judge:list": "eval-kit list-cases --config evals/eval-kit.model-judge.config.json", + "eval:judge:validate-fixtures": "eval-kit validate-fixtures --config evals/eval-kit.model-judge.config.json", + "eval:judge:coverage": "eval-kit judge-coverage --config evals/eval-kit.model-judge.config.json" +} +``` + +Keep `generate`, `judge_pairwise`, and `report` disabled in this manual config. Add pairwise later +with a separate config only after the consumer has calibrated pointwise evidence. + ## Review checklist ```text diff --git a/docs/guides/model-assisted-evals.md b/docs/guides/model-assisted-evals.md index e3798db..900d5c6 100644 --- a/docs/guides/model-assisted-evals.md +++ b/docs/guides/model-assisted-evals.md @@ -26,6 +26,33 @@ A consumer repo must provide: - result review process; - calibration policy before using judge results as gates outside default CI. +## Standard consumer config + +Use two configs for pointwise model judging: + +- `evals/eval-kit.config.json` is the default deterministic config. It is the only config that + belongs in `pnpm check` or CI. Keep `generate`, `judge_coverage`, and `judge_pairwise` disabled. +- `evals/eval-kit.model-judge.config.json` is the manual pointwise config. Enable + `judge_coverage`, disable `generate`, `judge_pairwise`, and `report`, and include only the cases + the consumer has chosen for manual judge calibration. + +Use stable script names so agents can run the right config without guessing: + +```json +{ + "eval:judge:doctor": "eval-kit doctor --config evals/eval-kit.model-judge.config.json", + "eval:judge:list": "eval-kit list-cases --config evals/eval-kit.model-judge.config.json", + "eval:judge:validate-fixtures": "eval-kit validate-fixtures --config evals/eval-kit.model-judge.config.json", + "eval:judge:coverage": "eval-kit judge-coverage --config evals/eval-kit.model-judge.config.json" +} +``` + +Do not ask contributors to temporarily flip `judge_coverage` in the deterministic config. That +creates drift between local manual runs and CI-safe defaults. + +Pairwise judging is a separate follow-up lane. Add `evals/eval-kit.pairwise.config.json` only after +the consumer has calibrated pointwise evidence and a specific comparison need. + ## Commands ```bash @@ -38,6 +65,7 @@ pnpm exec eval-kit generate \ pnpm exec eval-kit judge-coverage \ + --config evals/eval-kit.model-judge.config.json \ --case \ --candidate \ --model \ diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md index 8ba878a..14d37ab 100644 --- a/docs/guides/quickstart.md +++ b/docs/guides/quickstart.md @@ -7,7 +7,7 @@ This guide adds a generic deterministic eval suite to a consumer repo. ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" } } ``` diff --git a/docs/reference/release-process.md b/docs/reference/release-process.md index f581cc2..ef0378e 100644 --- a/docs/reference/release-process.md +++ b/docs/reference/release-process.md @@ -34,7 +34,7 @@ Consumers depend on tags like: Title: ```text -chore(release): v0.1.2 +chore(release): v0.1.3 ``` Required changes: @@ -63,18 +63,18 @@ git checkout main git pull --ff-only git rev-parse HEAD -git tag -a v0.1.2 -m "v0.1.2" -git push origin v0.1.2 +git tag -a v0.1.3 -m "v0.1.3" +git push origin v0.1.3 ``` Verify: ```bash -git rev-parse v0.1.2^{} -git show --no-patch --decorate v0.1.2 +git rev-parse v0.1.3^{} +git show --no-patch --decorate v0.1.3 ``` -`v0.1.2^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.2` +`v0.1.3^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.3` returns the tag object; `^{}` dereferences to the commit. ## GitHub Release @@ -93,7 +93,7 @@ For each consumer repo: ```json { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.2" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.3" } ``` @@ -108,7 +108,7 @@ pnpm check 3. Run consumer smoke commands, for example in `technical-design`: ```bash -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.2 +pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.3 ``` 4. Open a PR with dependency, lockfile, and any compatibility fixes. @@ -120,7 +120,7 @@ Do not move the tag. Create a new patch release: ```text -v0.1.2 -> v0.1.3 +v0.1.3 -> v0.1.4 ``` Then open consumer bump PRs. diff --git a/package.json b/package.json index 671003a..21d7d71 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@agentic-workflow-kit/eval-kit", - "version": "0.1.2", + "version": "0.1.3", "description": "Portable eval runner primitives for local eval suites.", "private": true, "type": "module", diff --git a/skills/bootstrap-eval-suite/SKILL.md b/skills/bootstrap-eval-suite/SKILL.md index 7c60391..33fd968 100644 --- a/skills/bootstrap-eval-suite/SKILL.md +++ b/skills/bootstrap-eval-suite/SKILL.md @@ -15,6 +15,18 @@ Use this skill when adding `@agentic-workflow-kit/eval-kit` harness files to a c 4. Run `pnpm exec eval-kit init --suite generic` only when the dry-run matches the intended scope. 5. Run `pnpm exec eval-kit doctor` and record the command result. +## Manual Model-Judge Config + +When adding pointwise model judging to a consumer that already has deterministic evals, use the +standard two-config pattern: + +- Keep `evals/eval-kit.config.json` deterministic and CI-safe, with model-assisted methods disabled. +- Add `evals/eval-kit.model-judge.config.json` for manual pointwise judging, with + `judge_coverage.enabled=true` and `generate`, `judge_pairwise`, and `report` disabled. +- Add scripts named `eval:judge:doctor`, `eval:judge:list`, `eval:judge:validate-fixtures`, and + `eval:judge:coverage` that point at the manual config. +- Keep pairwise out of this standard setup. Add a separate pairwise config only after calibration. + ## Boundaries - The kit owns loading, validation, discovery, result plumbing, and generic bootstrap mechanics. @@ -22,6 +34,7 @@ Use this skill when adding `@agentic-workflow-kit/eval-kit` harness files to a c - Do not add technical-design, define-product, design-to-plan, jig, or learning-loop semantics by default. - Do not install Promptfoo or require Codex auth for deterministic bootstrap. - Do not add run-producing, model-assisted, networked, or manually calibrated evals to default CI. +- Do not tell users to edit the deterministic config to enable manual judge runs. - Use `--force` only after inspecting existing files and deciding the overwrite is intended. ## Evidence diff --git a/skills/review-eval-suite/SKILL.md b/skills/review-eval-suite/SKILL.md index b1f5135..665823c 100644 --- a/skills/review-eval-suite/SKILL.md +++ b/skills/review-eval-suite/SKILL.md @@ -13,13 +13,17 @@ Use this skill when auditing or reviewing an eval-kit suite. 2. Run `pnpm exec eval-kit doctor` and treat failures as primary evidence. 3. Run `pnpm exec eval-kit list-cases` and inspect representative case manifests and artifacts. 4. Check whether generated or shared files contain consumer semantics that should stay local, or local files duplicate kit mechanics unnecessarily. -5. If reviewing a change, verify overwrite behavior and generated files with a dry run where relevant. +5. If model judging is present, confirm the default config stays deterministic and a separate manual + model-judge config owns pointwise `judge_coverage`. +6. If reviewing a change, verify overwrite behavior and generated files with a dry run where relevant. ## Boundaries - Mechanics belong in eval-kit; product vocabulary, rubric policy, cases, and expected outputs belong in the consumer. - Do not require Promptfoo for deterministic suites. - Treat model-assisted judging as advisory unless the consumer documents calibration and acceptance thresholds. +- Prefer `evals/eval-kit.model-judge.config.json` for manual pointwise judging; do not require users + to flip enabled flags in the deterministic config. - Treat run-producing semantic portfolios as local on-demand evidence before significant changes, not default CI. - Do not claim suite readiness without command evidence. diff --git a/skills/run-eval-suite/SKILL.md b/skills/run-eval-suite/SKILL.md index 9ac13f2..96e2661 100644 --- a/skills/run-eval-suite/SKILL.md +++ b/skills/run-eval-suite/SKILL.md @@ -14,7 +14,8 @@ Use this skill when executing a local eval-kit suite. 3. Run `pnpm exec eval-kit list-cases` and select the requested case ids or the repo-documented subset. 4. For deterministic runs, execute `pnpm exec eval-kit run-case --case --candidate ` for each selected case. 5. Compile reports with `pnpm exec eval-kit report` when the consumer workflow asks for a combined summary. -6. Run model-assisted commands only when explicitly requested and configured. +6. Run pointwise model-assisted commands only when explicitly requested, configured, and routed + through `evals/eval-kit.model-judge.config.json`. ## Boundaries @@ -24,8 +25,11 @@ Use this skill when executing a local eval-kit suite. - Do not require Codex auth or Promptfoo for deterministic-only runs. - Do not put Promptfoo provider calls, Codex/OpenAI calls, LLM judging, pairwise judging, or long replay suites in `pnpm check`. +- Use the consumer's `eval:judge:doctor`, `eval:judge:list`, and `eval:judge:validate-fixtures` + scripts before any manual `eval:judge:coverage` run. - Preserve raw outputs according to the consumer repo's artifact policy. ## Evidence -Report the config path, cases run, result directories, verdicts, report paths, and any skipped or advisory-only checks. +Report the config path, cases run, result directories, verdicts, report paths, and any skipped or +advisory-only checks. For model-assisted runs, state that provider calls were explicitly requested. diff --git a/src/sdk.mjs b/src/sdk.mjs index a625776..b935084 100644 --- a/src/sdk.mjs +++ b/src/sdk.mjs @@ -15,7 +15,7 @@ import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs"; const DEFAULT_SANDBOX_MODE = "read-only"; const DEFAULT_APPROVAL_POLICY = "never"; const RANDOMIZATION_METHOD = "sha256-seed-parity-v1"; -const EVAL_KIT_VERSION = "0.1.2"; +const EVAL_KIT_VERSION = "0.1.3"; const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");