docs(evals): align model-judge calibration policy

evals/model-judge/README.md

@@ -69,15 +69,19 @@ Start calibration with the committed good and bad candidates for the P0 cases:
 For each run, compare `pointwise-result.json` against the expected human label:
 
 - good candidates should have no critical `missing` or `contradicted` items;
-- bad candidates should expose the intended critical miss or contradiction;
+- `partial` is non-covered calibration evidence unless a non-critical item is explicitly accepted;
+- bad candidates should expose the intended critical miss, contradiction, partial coverage, or
+  ambiguity instead of receiving pass-like `covered` verdicts for the targeted defect;
 - `unknown` should be rare and should identify real ambiguity, not prompt weakness.
 
 Record curated observations here or in `evals/results/README.md`: false pass, false fail, ambiguity,
 verbosity bias, reference-wording bias, and any prompt or item wording adjustment. Model-judge output
 stays advisory even after calibration unless the repo policy changes explicitly.
 
-Use `evals/model-judge/calibration-notes.template.md` for human-reviewed calibration notes. Keep raw
-Promptfoo outputs local under `evals/results/`; commit only curated summaries.
+This repo follows the shared eval-kit `v0.1.5` calibration/reporting policy while keeping
+define-product PRD semantics local. Use `evals/model-judge/calibration-notes.template.md` for
+human-reviewed calibration notes. Keep raw Promptfoo outputs local under `evals/results/`; commit
+only curated summaries.
 
 ## Manual Reports
 
@@ -92,5 +96,6 @@ Report policy:
 
 - include deterministic evidence first;
 - include pointwise evidence only after deterministic evidence for the same review context exists;
+- include counts for `covered`, `partial`, `missing`, `contradicted`, and `unknown`;
 - state that model-judge results cannot upgrade deterministic red or yellow verdicts;
 - keep generated report bundles under ignored `evals/results/` unless a human curates a summary.

evals/model-judge/calibration-notes.template.md

@@ -17,13 +17,13 @@ Date: `<YYYY-MM-DD>`
 ## Expected Human Label
 
 - Good candidate expectation: `<critical items covered>`
-- Bad candidate expectation: `<intended critical misses or contradictions>`
+- Bad candidate expectation: `<intended critical misses, partials, contradictions, or ambiguity>`
 
 ## Item Review
 
-| Item ID | Expected human label | Judge verdict | Disposition            | Notes          |
-| ------- | -------------------- | ------------- | ---------------------- | -------------- |
-| `<id>`  | `<covered/missing>`  | `<verdict>`   | `<ok/fp/fn/ambiguous>` | `<short note>` |
+| Item ID | Expected human label             | Judge verdict | Disposition            | Notes          |
+| ------- | -------------------------------- | ------------- | ---------------------- | -------------- |
+| `<id>`  | `<covered/adverse/non-critical>` | `<verdict>`   | `<ok/fp/fn/ambiguous>` | `<short note>` |
 
 ## Bias And Failure Modes
 
@@ -33,6 +33,8 @@ Date: `<YYYY-MM-DD>`
 - Verbosity bias:
 - Reference-wording bias:
 - Unknown rate:
+- Critical partials:
+- Expected-bad pass-like coverage:
 
 ## Decision
 

evals/results/README.md

@@ -6,3 +6,8 @@ only; transient run directories stay ignored by default.
 Manual report bundles may include deterministic grades and pointwise model-judge results copied from
 local runs. Treat them as local evidence unless a human reviews and curates a summary. Raw Promptfoo
 outputs, provider metadata, candidate text, and generated HTML reports should stay uncommitted.
+
+Read manual pointwise verdicts with the eval-kit `v0.1.5` calibration policy: `partial`, `missing`,
+`contradicted`, and `unknown` are non-covered evidence unless a curated define-product note says a
+non-critical partial is acceptable. Bad PRD fixtures are expected to remain adverse on their targeted
+defect; pass-like `covered` verdicts there are false-pass risks.

package.json

@@ -35,7 +35,7 @@
     "eval:report": "eval-kit report --config evals/eval-kit.config.json"
   },
   "devDependencies": {
-    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.4",
+    "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5",
     "prettier": "^3.6.2",
     "promptfoo": "^0.121.17"
   }