docs: standardize model-judge calibration reporting

CHANGELOG.md

@@ -12,6 +12,22 @@ documented with migration notes.
 - Additional docs for suite-specific adoption.
 - Better compatibility tests for Promptfoo variable contracts.
 
+## [0.1.5] - 2026-07-04
+
+### Added
+
+- Documented shared manual pointwise model-judge calibration and reporting policy.
+- Clarified `covered`, `partial`, `missing`, `contradicted`, and `unknown` interpretation.
+- Updated eval-kit skills to report deterministic evidence first, keep raw provider bundles local
+  unless curated, and treat expected-bad fixtures as successful calibration evidence when they
+  remain adverse.
+
+### Notes
+
+- No runtime, schema, prompt, or CLI behavior changed.
+- No npm package is published.
+- Consumers may pin `github:agentic-workflow-kit/eval-kit#v0.1.5`.
+
 ## [0.1.4] - 2026-07-04
 
 ### Fixed
@@ -91,7 +107,8 @@ documented with migration notes.
 - Suite-specific presets remain deferred.
 - Consumer repos own their own semantics, prompts, cases, and pass/fail policies.
 
-[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.4...main
+[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.5...main
+[0.1.5]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.4...v0.1.5
 [0.1.4]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.3...v0.1.4
 [0.1.3]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.1...v0.1.2

README.md

@@ -11,7 +11,7 @@ Shared evaluation infrastructure for `agentic-workflow-kit` repositories.
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
   }
 }
 ```
@@ -70,7 +70,7 @@ Install from a Git tag in a consumer repo:
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
   },
   "scripts": {
     "eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json",
@@ -156,6 +156,7 @@ Start here:
 - [`docs/guides/quickstart.md`](docs/guides/quickstart.md) - bootstrap and first deterministic run.
 - [`docs/guides/consumer-integration.md`](docs/guides/consumer-integration.md) - add eval-kit to another repo.
 - [`docs/guides/model-assisted-evals.md`](docs/guides/model-assisted-evals.md) - configure manual model-judge lanes.
+- [`docs/guides/model-judge-calibration-reporting.md`](docs/guides/model-judge-calibration-reporting.md) - interpret and report manual pointwise evidence.
 - [`docs/reference/adapter-contract.md`](docs/reference/adapter-contract.md) - adapter exports and hook shapes.
 - [`docs/reference/release-process.md`](docs/reference/release-process.md) - version and tag process.
 
@@ -191,7 +192,7 @@ v0.1.0
 v0.1.1
 v0.1.2
 v0.1.3
-v0.1.4
+v0.1.5
 v0.2.0
 ```
 

docs/design/consumer-integration.md

@@ -9,7 +9,7 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
   }
 }
 ```

docs/guides/consumer-integration.md

@@ -18,7 +18,7 @@ If you cannot state the eval goal, do not bootstrap a suite yet. Empty harnesses
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
   }
 }
 ```
@@ -106,5 +106,10 @@ with a separate config only after the consumer has calibrated pointwise evidence
 - [ ] Are hidden reference answers avoided?
 - [ ] Are deterministic blockers not averaged away?
 - [ ] Are model judges advisory unless calibrated?
+- [ ] Are expected-bad fixtures expected to remain adverse instead of receiving pass-like coverage?
+- [ ] Are raw provider result bundles ignored unless a human curates a summary?
 - [ ] Does `pnpm check` remain fast, offline, and reasonable for contributors?
 ```
+
+For manual pointwise calibration and report wording, follow
+[`model-judge-calibration-reporting.md`](model-judge-calibration-reporting.md).

docs/guides/model-assisted-evals.md

@@ -24,7 +24,7 @@ A consumer repo must provide:
 - adapter hooks for prompt variables;
 - rubrics;
 - result review process;
-- calibration policy before using judge results as gates outside default CI.
+- calibration policy before treating judge results as anything more than advisory evidence.
 
 ## Standard consumer config
 
@@ -154,10 +154,35 @@ Model judge prompts should:
 
 ## Calibration
 
-Before using model judges as gates, keep a human-reviewed golden set and track:
+Calibration starts from committed expected-good and expected-bad fixtures. A judge run is useful
+only when a human reviewer can explain whether each verdict matched the fixture intent.
+
+Pointwise verdicts have these default meanings:
+
+- `covered`: the candidate clearly satisfies the item with cited evidence.
+- `partial`: the candidate has some relevant evidence but does not fully satisfy the item. Treat it
+  as non-covered unless the consumer policy explicitly accepts that item as non-critical.
+- `missing`: the candidate lacks enough evidence for the item.
+- `contradicted`: the candidate conflicts with the expected item.
+- `unknown`: the visible evidence is too ambiguous to judge. Track this as ambiguity or prompt/item
+  weakness until reviewed.
+
+For expected-good fixtures, critical items should be `covered`; any critical `partial`, `missing`,
+`contradicted`, or repeated `unknown` needs human review. For expected-bad fixtures, adverse
+verdicts such as `partial`, `missing`, `contradicted`, or `unknown` are expected when they match the
+fixture's intended defect. Bad fixtures should not receive pass-like `covered` verdicts for the
+targeted critical defect.
+
+Before using model judges as review evidence, keep a human-reviewed golden set and track:
 
 - false passes on invented facts;
 - false failures on acceptable alternatives;
+- expected-bad fixtures that receive pass-like coverage;
+- critical `partial` or `unknown` verdicts on expected-good fixtures;
 - position bias in pairwise comparison;
 - verbosity bias;
 - over-reliance on reference wording.
+
+Raw Promptfoo and provider outputs should stay under ignored `evals/results/` paths unless a human
+curates a summary. Reports should present deterministic verdicts first and state that model-judge
+results cannot upgrade deterministic blockers.

docs/guides/model-judge-calibration-reporting.md

@@ -0,0 +1,47 @@
+# Model-judge calibration and reporting
+
+Manual model-judge evidence is useful only after a human reviewer checks it against committed
+calibration fixtures. It remains advisory unless a consumer repo explicitly changes its own policy.
+
+## Verdict interpretation
+
+Use the same pointwise meaning across consumers:
+
+- `covered`: the candidate clearly satisfies the item with direct candidate evidence.
+- `partial`: the candidate addresses part of the item but does not fully satisfy it. Treat it as
+  non-covered unless the consumer explicitly accepts that non-critical item.
+- `missing`: the candidate does not provide enough support for the item.
+- `contradicted`: the candidate conflicts with the item.
+- `unknown`: the candidate is too ambiguous to judge from visible inputs.
+
+`partial`, `missing`, `contradicted`, and `unknown` are adverse evidence for a target item unless the
+consumer's documented policy says otherwise.
+
+## Good and bad fixtures
+
+Calibrate with committed expected-good and expected-bad fixtures:
+
+- Expected-good fixtures should cover critical items. A critical `partial`, `missing`,
+  `contradicted`, or repeated `unknown` is a calibration issue until a human explains it.
+- Expected-bad fixtures should remain adverse on the intended defect. That means the targeted
+  critical item should not receive pass-like `covered` evidence.
+- A bad fixture is behaving correctly when the judge identifies the intended miss, contradiction, or
+  ambiguity.
+
+When a bad fixture receives `covered` for the targeted defect, classify it as a false pass. When a
+good fixture receives an unjustified adverse verdict, classify it as a false fail.
+
+## Reporting
+
+Manual reports should be written for reviewer handoff, not CI:
+
+- show deterministic evidence first;
+- show model-judge evidence as a separate advisory section;
+- include verdict counts for `covered`, `partial`, `missing`, `contradicted`, and `unknown`;
+- identify expected-good and expected-bad fixture labels;
+- record false pass, false fail, ambiguity, verbosity bias, reference-wording bias, and unknown-rate
+  risks;
+- state that model-judge evidence cannot upgrade deterministic red or yellow results.
+
+Keep raw provider bundles under ignored `evals/results/` paths unless a human curates and commits a
+summary.

docs/guides/quickstart.md

@@ -7,7 +7,7 @@ This guide adds a generic deterministic eval suite to a consumer repo.
 ```json
 {
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
   }
 }
 ```

docs/reference/release-process.md

@@ -34,7 +34,7 @@ Consumers depend on tags like:
 Title:
 
 ```text
-chore(release): v0.1.4
+chore(release): v0.1.5
 ```
 
 Required changes:
@@ -63,18 +63,18 @@ git checkout main
 git pull --ff-only
 git rev-parse HEAD
 
-git tag -a v0.1.4 -m "v0.1.4"
-git push origin v0.1.4
+git tag -a v0.1.5 -m "v0.1.5"
+git push origin v0.1.5
 ```
 
 Verify:
 
 ```bash
-git rev-parse v0.1.4^{}
-git show --no-patch --decorate v0.1.4
+git rev-parse v0.1.5^{}
+git show --no-patch --decorate v0.1.5
 ```
 
-`v0.1.4^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.4`
+`v0.1.5^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.5`
 returns the tag object; `^{}` dereferences to the commit.
 
 ## GitHub Release
@@ -93,7 +93,7 @@ For each consumer repo:
 
 ```json
 {
-  "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4"
+  "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
 }
 ```
 
@@ -108,7 +108,7 @@ pnpm check
 3. Run consumer smoke commands, for example in `technical-design`:
 
 ```bash
-pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.4
+pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.5
 ```
 
 4. Open a PR with dependency, lockfile, and any compatibility fixes.
@@ -120,7 +120,7 @@ Do not move the tag.
 Create a new patch release:
 
 ```text
-v0.1.4 -> v0.1.5
+v0.1.5 -> v0.1.6
 ```
 
 Then open consumer bump PRs.

docs/reference/results.md

@@ -39,7 +39,7 @@ Current schema:
   "run_type": "deterministic",
   "runner": {
     "id": "generic-eval-case",
-    "version": "0.1.4"
+    "version": "0.1.5"
   },
   "case_ids": ["case-example-v1"],
   "started_at": "2026-07-03T00:00:00.000Z",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agentic-workflow-kit/eval-kit",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Portable eval runner primitives for local eval suites.",
   "private": true,
   "type": "module",

skills/bootstrap-eval-suite/SKILL.md

@@ -26,6 +26,9 @@ standard two-config pattern:
 - Add scripts named `eval:judge:doctor`, `eval:judge:list`, `eval:judge:validate-fixtures`, and
   `eval:judge:coverage` that point at the manual config.
 - Keep pairwise out of this standard setup. Add a separate pairwise config only after calibration.
+- Document the local calibration policy before treating pointwise results as more than raw advisory
+  evidence. The policy should define expected-good and expected-bad fixture labels, `partial` and
+  `unknown` handling, and where curated summaries live.
 
 ## Boundaries
 
@@ -35,6 +38,7 @@ standard two-config pattern:
 - Do not install Promptfoo or require Codex auth for deterministic bootstrap.
 - Do not add run-producing, model-assisted, networked, or manually calibrated evals to default CI.
 - Do not tell users to edit the deterministic config to enable manual judge runs.
+- Do not commit raw provider result bundles unless a human has curated them according to repo policy.
 - Use `--force` only after inspecting existing files and deciding the overwrite is intended.
 
 ## Evidence

skills/review-eval-suite/SKILL.md

@@ -24,9 +24,15 @@ Use this skill when auditing or reviewing an eval-kit suite.
 - Treat model-assisted judging as advisory unless the consumer documents calibration and acceptance thresholds.
 - Prefer `evals/eval-kit.model-judge.config.json` for manual pointwise judging; do not require users
   to flip enabled flags in the deterministic config.
+- Review expected-good and expected-bad calibration separately. Expected-bad fixtures should produce
+  adverse evidence on the targeted defect; pass-like `covered` verdicts for that defect are false
+  passes.
+- Treat `partial` as non-covered unless the consumer explicitly documents why a non-critical partial
+  is acceptable. Repeated `unknown` verdicts are calibration or prompt-quality risks.
 - Treat run-producing semantic portfolios as local on-demand evidence before significant changes, not default CI.
 - Do not claim suite readiness without command evidence.
 
 ## Evidence
 
-Report command results, reviewed case ids, concrete boundary issues, and remaining risks.
+Report command results, reviewed case ids, concrete boundary issues, calibration false pass/false
+fail risks, `partial`/`unknown` interpretation, and remaining risks.

skills/run-eval-suite/SKILL.md

@@ -27,9 +27,15 @@ Use this skill when executing a local eval-kit suite.
   replay suites in `pnpm check`.
 - Use the consumer's `eval:judge:doctor`, `eval:judge:list`, and `eval:judge:validate-fixtures`
   scripts before any manual `eval:judge:coverage` run.
+- For pointwise model-judge summaries, treat `partial`, `missing`, `contradicted`, and `unknown` as
+  non-covered unless the consumer policy explicitly accepts the item.
+- Expected-bad fixtures should remain adverse on their intended defect. Do not describe an adverse
+  bad-fixture result as a failed eval when it matches the calibration label.
 - Preserve raw outputs according to the consumer repo's artifact policy.
 
 ## Evidence
 
 Report the config path, cases run, result directories, verdicts, report paths, and any skipped or
 advisory-only checks. For model-assisted runs, state that provider calls were explicitly requested.
+Report deterministic evidence first, then model-judge counts for `covered`, `partial`, `missing`,
+`contradicted`, and `unknown`.

src/sdk.mjs

@@ -15,7 +15,7 @@ import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs";
 const DEFAULT_SANDBOX_MODE = "read-only";
 const DEFAULT_APPROVAL_POLICY = "never";
 const RANDOMIZATION_METHOD = "sha256-seed-parity-v1";
-const EVAL_KIT_VERSION = "0.1.4";
+const EVAL_KIT_VERSION = "0.1.5";
 
 const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");