docs(evals): align model-judge calibration policy

docs/design/evaluation-strategy.md

@@ -302,6 +302,11 @@ The judge must grade against a rubric and cite evidence. It must be allowed to r
 It must not reward length, rhetorical confidence, or familiar architecture vocabulary without
 source support.
 
+Pointwise verdicts use the shared eval-kit `v0.1.5` calibration policy. `partial`, `missing`,
+`contradicted`, and `unknown` are non-covered evidence unless a curated repo-local note accepts a
+non-critical partial. Expected-bad fixtures should remain adverse on the intended defect; pass-like
+`covered` verdicts there are false-pass risks.
+
 Run pointwise coverage judging before pairwise comparison when inspecting generated case outputs.
 The pointwise judge grades each expected `FACT-*` and `CTX-*` item against candidate evidence and
 visible source inputs. Pairwise judging answers which candidate is better overall; it does not prove
@@ -348,6 +353,8 @@ Calibration should track:
 
 - false passes on invented facts;
 - false failures on defensible alternative designs;
+- critical `partial` or repeated `unknown` verdicts on expected-good fixtures;
+- pass-like coverage on the targeted defect in expected-bad fixtures;
 - position bias in pairwise judging;
 - preference for verbosity over clarity;
 - over-reliance on reference design wording.

evals/README.md

@@ -116,6 +116,12 @@ This repo follows the eval-kit two-config standard:
   It disables generation, pairwise judging, and reports.
 - Pairwise judging remains deferred. Add a separate `evals/eval-kit.pairwise.config.json` only if a
   future calibrated comparison workflow needs it.
+- Manual pointwise calibration follows the shared eval-kit `v0.1.5` calibration/reporting policy
+  while this repo keeps technical-design expected facts, boundaries, and rubric semantics local.
+  Treat `partial`, `missing`, `contradicted`, and `unknown` as non-covered evidence unless a curated
+  note explicitly accepts a non-critical partial.
+- Expected-bad fixtures, when added, should remain adverse on their targeted defect. Pass-like
+  `covered` verdicts for those targeted defects are false-pass risks.
 
 Check local auth:
 

evals/implementation-plan.md

@@ -299,13 +299,17 @@ Integration point:
 - Judge runs remain manual/advisory. They may inform manual review and future deterministic/rubric
   changes after calibration, but they cannot override deterministic blockers or become automatic PR
   blockers.
+- Pointwise `partial`, `missing`, `contradicted`, and `unknown` verdicts are non-covered evidence
+  unless a curated technical-design calibration note accepts a non-critical partial. Expected-bad
+  fixtures should remain adverse on their targeted defect instead of receiving pass-like coverage.
 - Pairwise regression remains deferred until pointwise calibration shows a specific comparison need.
 
 Verification:
 
 - Deterministic JSON parse and schema validation of every judge result.
 - Pairwise order randomization proof in run metadata.
-- Human calibration agreement report under `evals/results/<run-id>/`.
+- Human calibration agreement report as a curated summary; raw `evals/results/<run-id>/` bundles stay
+  local unless a reviewer promotes a redacted summary.
 
 ## Phase 5 - Outcome Studies
 

evals/results/README.md

@@ -41,3 +41,7 @@ How to read a result bundle:
    present.
 5. Open `promptfoo-results.json` only when debugging provider output or cost/runtime metadata.
 6. Open `cases/<case-id>/candidate.md` only when a finding needs source evidence.
+
+Manual pointwise summaries follow the eval-kit `v0.1.5` calibration policy: show deterministic
+evidence first, report counts for `covered`, `partial`, `missing`, `contradicted`, and `unknown`,
+and keep raw provider bundles local unless a human curates a redacted summary.

package.json

@@ -48,7 +48,7 @@
     "eval:enforce": "node scripts/run_enforce_eval.mjs"
   },
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4",
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5",
     "ajv": "^8.17.1",
     "dependency-cruiser": "^18.0.0",
     "prettier": "^3.6.2",