diff --git a/CHANGELOG.md b/CHANGELOG.md index 62f6c90..dd1d45a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,22 @@ documented with migration notes. - Additional docs for suite-specific adoption. - Better compatibility tests for Promptfoo variable contracts. +## [0.1.5] - 2026-07-04 + +### Added + +- Documented shared manual pointwise model-judge calibration and reporting policy. +- Clarified `covered`, `partial`, `missing`, `contradicted`, and `unknown` interpretation. +- Updated eval-kit skills to report deterministic evidence first, keep raw provider bundles local + unless curated, and treat expected-bad fixtures as successful calibration evidence when they + remain adverse. + +### Notes + +- No runtime, schema, prompt, or CLI behavior changed. +- No npm package is published. +- Consumers may pin `github:agentic-workflow-kit/eval-kit#v0.1.5`. + ## [0.1.4] - 2026-07-04 ### Fixed @@ -91,7 +107,8 @@ documented with migration notes. - Suite-specific presets remain deferred. - Consumer repos own their own semantics, prompts, cases, and pass/fail policies. -[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.4...main +[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.5...main +[0.1.5]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.4...v0.1.5 [0.1.4]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.3...v0.1.4 [0.1.3]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...v0.1.3 [0.1.2]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.1...v0.1.2 diff --git a/README.md b/README.md index 5819f13..1e29e03 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Shared evaluation infrastructure for `agentic-workflow-kit` repositories. ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" } } ``` @@ -70,7 +70,7 @@ Install from a Git tag in a consumer repo: ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" }, "scripts": { "eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json", @@ -156,6 +156,7 @@ Start here: - [`docs/guides/quickstart.md`](docs/guides/quickstart.md) - bootstrap and first deterministic run. - [`docs/guides/consumer-integration.md`](docs/guides/consumer-integration.md) - add eval-kit to another repo. - [`docs/guides/model-assisted-evals.md`](docs/guides/model-assisted-evals.md) - configure manual model-judge lanes. +- [`docs/guides/model-judge-calibration-reporting.md`](docs/guides/model-judge-calibration-reporting.md) - interpret and report manual pointwise evidence. - [`docs/reference/adapter-contract.md`](docs/reference/adapter-contract.md) - adapter exports and hook shapes. - [`docs/reference/release-process.md`](docs/reference/release-process.md) - version and tag process. @@ -191,7 +192,7 @@ v0.1.0 v0.1.1 v0.1.2 v0.1.3 -v0.1.4 +v0.1.5 v0.2.0 ``` diff --git a/docs/design/consumer-integration.md b/docs/design/consumer-integration.md index 617a9b1..2f02653 100644 --- a/docs/design/consumer-integration.md +++ b/docs/design/consumer-integration.md @@ -9,7 +9,7 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" } } ``` diff --git a/docs/guides/consumer-integration.md b/docs/guides/consumer-integration.md index bffde6b..4b8d3ce 100644 --- a/docs/guides/consumer-integration.md +++ b/docs/guides/consumer-integration.md @@ -18,7 +18,7 @@ If you cannot state the eval goal, do not bootstrap a suite yet. Empty harnesses ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" } } ``` @@ -106,5 +106,10 @@ with a separate config only after the consumer has calibrated pointwise evidence - [ ] Are hidden reference answers avoided? - [ ] Are deterministic blockers not averaged away? - [ ] Are model judges advisory unless calibrated? +- [ ] Are expected-bad fixtures expected to remain adverse instead of receiving pass-like coverage? +- [ ] Are raw provider result bundles ignored unless a human curates a summary? - [ ] Does `pnpm check` remain fast, offline, and reasonable for contributors? ``` + +For manual pointwise calibration and report wording, follow +[`model-judge-calibration-reporting.md`](model-judge-calibration-reporting.md). diff --git a/docs/guides/model-assisted-evals.md b/docs/guides/model-assisted-evals.md index e49b09f..35ae375 100644 --- a/docs/guides/model-assisted-evals.md +++ b/docs/guides/model-assisted-evals.md @@ -24,7 +24,7 @@ A consumer repo must provide: - adapter hooks for prompt variables; - rubrics; - result review process; -- calibration policy before using judge results as gates outside default CI. +- calibration policy before treating judge results as anything more than advisory evidence. ## Standard consumer config @@ -154,10 +154,35 @@ Model judge prompts should: ## Calibration -Before using model judges as gates, keep a human-reviewed golden set and track: +Calibration starts from committed expected-good and expected-bad fixtures. A judge run is useful +only when a human reviewer can explain whether each verdict matched the fixture intent. + +Pointwise verdicts have these default meanings: + +- `covered`: the candidate clearly satisfies the item with cited evidence. +- `partial`: the candidate has some relevant evidence but does not fully satisfy the item. Treat it + as non-covered unless the consumer policy explicitly accepts that item as non-critical. +- `missing`: the candidate lacks enough evidence for the item. +- `contradicted`: the candidate conflicts with the expected item. +- `unknown`: the visible evidence is too ambiguous to judge. Track this as ambiguity or prompt/item + weakness until reviewed. + +For expected-good fixtures, critical items should be `covered`; any critical `partial`, `missing`, +`contradicted`, or repeated `unknown` needs human review. For expected-bad fixtures, adverse +verdicts such as `partial`, `missing`, `contradicted`, or `unknown` are expected when they match the +fixture's intended defect. Bad fixtures should not receive pass-like `covered` verdicts for the +targeted critical defect. + +Before using model judges as review evidence, keep a human-reviewed golden set and track: - false passes on invented facts; - false failures on acceptable alternatives; +- expected-bad fixtures that receive pass-like coverage; +- critical `partial` or `unknown` verdicts on expected-good fixtures; - position bias in pairwise comparison; - verbosity bias; - over-reliance on reference wording. + +Raw Promptfoo and provider outputs should stay under ignored `evals/results/` paths unless a human +curates a summary. Reports should present deterministic verdicts first and state that model-judge +results cannot upgrade deterministic blockers. diff --git a/docs/guides/model-judge-calibration-reporting.md b/docs/guides/model-judge-calibration-reporting.md new file mode 100644 index 0000000..4c5cf80 --- /dev/null +++ b/docs/guides/model-judge-calibration-reporting.md @@ -0,0 +1,47 @@ +# Model-judge calibration and reporting + +Manual model-judge evidence is useful only after a human reviewer checks it against committed +calibration fixtures. It remains advisory unless a consumer repo explicitly changes its own policy. + +## Verdict interpretation + +Use the same pointwise meaning across consumers: + +- `covered`: the candidate clearly satisfies the item with direct candidate evidence. +- `partial`: the candidate addresses part of the item but does not fully satisfy it. Treat it as + non-covered unless the consumer explicitly accepts that non-critical item. +- `missing`: the candidate does not provide enough support for the item. +- `contradicted`: the candidate conflicts with the item. +- `unknown`: the candidate is too ambiguous to judge from visible inputs. + +`partial`, `missing`, `contradicted`, and `unknown` are adverse evidence for a target item unless the +consumer's documented policy says otherwise. + +## Good and bad fixtures + +Calibrate with committed expected-good and expected-bad fixtures: + +- Expected-good fixtures should cover critical items. A critical `partial`, `missing`, + `contradicted`, or repeated `unknown` is a calibration issue until a human explains it. +- Expected-bad fixtures should remain adverse on the intended defect. That means the targeted + critical item should not receive pass-like `covered` evidence. +- A bad fixture is behaving correctly when the judge identifies the intended miss, contradiction, or + ambiguity. + +When a bad fixture receives `covered` for the targeted defect, classify it as a false pass. When a +good fixture receives an unjustified adverse verdict, classify it as a false fail. + +## Reporting + +Manual reports should be written for reviewer handoff, not CI: + +- show deterministic evidence first; +- show model-judge evidence as a separate advisory section; +- include verdict counts for `covered`, `partial`, `missing`, `contradicted`, and `unknown`; +- identify expected-good and expected-bad fixture labels; +- record false pass, false fail, ambiguity, verbosity bias, reference-wording bias, and unknown-rate + risks; +- state that model-judge evidence cannot upgrade deterministic red or yellow results. + +Keep raw provider bundles under ignored `evals/results/` paths unless a human curates and commits a +summary. diff --git a/docs/guides/quickstart.md b/docs/guides/quickstart.md index a6a0fbc..34affea 100644 --- a/docs/guides/quickstart.md +++ b/docs/guides/quickstart.md @@ -7,7 +7,7 @@ This guide adds a generic deterministic eval suite to a consumer repo. ```json { "devDependencies": { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" } } ``` diff --git a/docs/reference/release-process.md b/docs/reference/release-process.md index d427df0..9e56429 100644 --- a/docs/reference/release-process.md +++ b/docs/reference/release-process.md @@ -34,7 +34,7 @@ Consumers depend on tags like: Title: ```text -chore(release): v0.1.4 +chore(release): v0.1.5 ``` Required changes: @@ -63,18 +63,18 @@ git checkout main git pull --ff-only git rev-parse HEAD -git tag -a v0.1.4 -m "v0.1.4" -git push origin v0.1.4 +git tag -a v0.1.5 -m "v0.1.5" +git push origin v0.1.5 ``` Verify: ```bash -git rev-parse v0.1.4^{} -git show --no-patch --decorate v0.1.4 +git rev-parse v0.1.5^{} +git show --no-patch --decorate v0.1.5 ``` -`v0.1.4^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.4` +`v0.1.5^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.5` returns the tag object; `^{}` dereferences to the commit. ## GitHub Release @@ -93,7 +93,7 @@ For each consumer repo: ```json { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5" } ``` @@ -108,7 +108,7 @@ pnpm check 3. Run consumer smoke commands, for example in `technical-design`: ```bash -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.4 +pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.5 ``` 4. Open a PR with dependency, lockfile, and any compatibility fixes. @@ -120,7 +120,7 @@ Do not move the tag. Create a new patch release: ```text -v0.1.4 -> v0.1.5 +v0.1.5 -> v0.1.6 ``` Then open consumer bump PRs. diff --git a/docs/reference/results.md b/docs/reference/results.md index df1a78a..edbdcd2 100644 --- a/docs/reference/results.md +++ b/docs/reference/results.md @@ -39,7 +39,7 @@ Current schema: "run_type": "deterministic", "runner": { "id": "generic-eval-case", - "version": "0.1.4" + "version": "0.1.5" }, "case_ids": ["case-example-v1"], "started_at": "2026-07-03T00:00:00.000Z", diff --git a/package.json b/package.json index 0829e31..a6da2f5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@agentic-workflow-kit/eval-kit", - "version": "0.1.4", + "version": "0.1.5", "description": "Portable eval runner primitives for local eval suites.", "private": true, "type": "module", diff --git a/skills/bootstrap-eval-suite/SKILL.md b/skills/bootstrap-eval-suite/SKILL.md index 33fd968..cc0db07 100644 --- a/skills/bootstrap-eval-suite/SKILL.md +++ b/skills/bootstrap-eval-suite/SKILL.md @@ -26,6 +26,9 @@ standard two-config pattern: - Add scripts named `eval:judge:doctor`, `eval:judge:list`, `eval:judge:validate-fixtures`, and `eval:judge:coverage` that point at the manual config. - Keep pairwise out of this standard setup. Add a separate pairwise config only after calibration. +- Document the local calibration policy before treating pointwise results as more than raw advisory + evidence. The policy should define expected-good and expected-bad fixture labels, `partial` and + `unknown` handling, and where curated summaries live. ## Boundaries @@ -35,6 +38,7 @@ standard two-config pattern: - Do not install Promptfoo or require Codex auth for deterministic bootstrap. - Do not add run-producing, model-assisted, networked, or manually calibrated evals to default CI. - Do not tell users to edit the deterministic config to enable manual judge runs. +- Do not commit raw provider result bundles unless a human has curated them according to repo policy. - Use `--force` only after inspecting existing files and deciding the overwrite is intended. ## Evidence diff --git a/skills/review-eval-suite/SKILL.md b/skills/review-eval-suite/SKILL.md index 665823c..b72d69e 100644 --- a/skills/review-eval-suite/SKILL.md +++ b/skills/review-eval-suite/SKILL.md @@ -24,9 +24,15 @@ Use this skill when auditing or reviewing an eval-kit suite. - Treat model-assisted judging as advisory unless the consumer documents calibration and acceptance thresholds. - Prefer `evals/eval-kit.model-judge.config.json` for manual pointwise judging; do not require users to flip enabled flags in the deterministic config. +- Review expected-good and expected-bad calibration separately. Expected-bad fixtures should produce + adverse evidence on the targeted defect; pass-like `covered` verdicts for that defect are false + passes. +- Treat `partial` as non-covered unless the consumer explicitly documents why a non-critical partial + is acceptable. Repeated `unknown` verdicts are calibration or prompt-quality risks. - Treat run-producing semantic portfolios as local on-demand evidence before significant changes, not default CI. - Do not claim suite readiness without command evidence. ## Evidence -Report command results, reviewed case ids, concrete boundary issues, and remaining risks. +Report command results, reviewed case ids, concrete boundary issues, calibration false pass/false +fail risks, `partial`/`unknown` interpretation, and remaining risks. diff --git a/skills/run-eval-suite/SKILL.md b/skills/run-eval-suite/SKILL.md index 96e2661..56da791 100644 --- a/skills/run-eval-suite/SKILL.md +++ b/skills/run-eval-suite/SKILL.md @@ -27,9 +27,15 @@ Use this skill when executing a local eval-kit suite. replay suites in `pnpm check`. - Use the consumer's `eval:judge:doctor`, `eval:judge:list`, and `eval:judge:validate-fixtures` scripts before any manual `eval:judge:coverage` run. +- For pointwise model-judge summaries, treat `partial`, `missing`, `contradicted`, and `unknown` as + non-covered unless the consumer policy explicitly accepts the item. +- Expected-bad fixtures should remain adverse on their intended defect. Do not describe an adverse + bad-fixture result as a failed eval when it matches the calibration label. - Preserve raw outputs according to the consumer repo's artifact policy. ## Evidence Report the config path, cases run, result directories, verdicts, report paths, and any skipped or advisory-only checks. For model-assisted runs, state that provider calls were explicitly requested. +Report deterministic evidence first, then model-judge counts for `covered`, `partial`, `missing`, +`contradicted`, and `unknown`. diff --git a/src/sdk.mjs b/src/sdk.mjs index 7fd3521..8d56b56 100644 --- a/src/sdk.mjs +++ b/src/sdk.mjs @@ -15,7 +15,7 @@ import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs"; const DEFAULT_SANDBOX_MODE = "read-only"; const DEFAULT_APPROVAL_POLICY = "never"; const RANDOMIZATION_METHOD = "sha256-seed-parity-v1"; -const EVAL_KIT_VERSION = "0.1.4"; +const EVAL_KIT_VERSION = "0.1.5"; const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");