docs(evals): add model-judge calibration notes

evals/model-judge/calibration-notes.md

@@ -0,0 +1,41 @@
+# Model-Judge Calibration Notes
+
+This file is the curated, human-readable calibration summary for the define-product manual pointwise model-judge lane. Raw provider result bundles under `evals/results/**` remain local/ignored and are not committed.
+
+Deterministic eval verdicts remain authoritative. Pointwise model-judge evidence is manual, advisory, and cannot upgrade deterministic blockers.
+
+## Run Set
+
+These notes summarize the existing local provider-backed runs from July 4, 2026. No fresh provider calls were made for this curated summary.
+
+| Field             | Value                                          |
+| ----------------- | ---------------------------------------------- |
+| Config            | `evals/eval-kit.model-judge.config.json`       |
+| Model             | `gpt-5.4`                                      |
+| Provider          | `openai:codex-app-server`                      |
+| Reasoning effort  | `medium`                                       |
+| Prompt version    | `define-product-pointwise-v1`                  |
+| Rubric version    | `define-product-pointwise-rubric-v1`           |
+| Candidate profile | P0 expected-good and expected-bad PRD fixtures |
+
+## Calibration Summary
+
+| Case                                | Candidate     | Run id                                                           | Covered | Partial | Missing | Contradicted | Unknown | Human disposition                                                                                           |
+| ----------------------------------- | ------------- | ---------------------------------------------------------------- | ------: | ------: | ------: | -----------: | ------: | ----------------------------------------------------------------------------------------------------------- |
+| `case-contract-grounded-prd-v1`     | expected-good | `provider-20260704-v015b-case-contract-grounded-prd-v1-good`     |       5 |       0 |       0 |            0 |       0 | Expected pass-like coverage on product grounding, acceptance criteria, uncertainty, and citation integrity. |
+| `case-contract-grounded-prd-v1`     | expected-bad  | `provider-20260704-v015b-case-contract-grounded-prd-v1-bad`      |       0 |       0 |       0 |            5 |       0 | Expected adverse evidence; no pass-like coverage on intended defects.                                       |
+| `case-altitude-and-future-scope-v1` | expected-good | `provider-20260704-v015b-case-altitude-and-future-scope-v1-good` |       4 |       0 |       0 |            0 |       0 | Expected pass-like coverage on altitude discipline, future scope, acceptance criteria, and citations.       |
+| `case-altitude-and-future-scope-v1` | expected-bad  | `provider-20260704-v015b-case-altitude-and-future-scope-v1-bad`  |       0 |       0 |       0 |            4 |       0 | Expected adverse evidence; no pass-like coverage on intended defects.                                       |
+
+## Human Calibration Notes
+
+- False pass risk: no false-pass evidence observed in the P0 expected-bad runs.
+- False fail risk: no false-fail evidence observed in the P0 expected-good runs.
+- Ambiguity: no `partial` or `unknown` verdicts appeared in these runs.
+- Verbosity bias: no evidence that verbose PRD wording was required for coverage.
+- Reference wording bias: current evidence is based on local committed fixtures; future expansions should continue to check for semantic coverage rather than reference-text matching.
+- Unknown rate: zero unknown verdicts in the summarized run set.
+
+## Decision
+
+Keep the define-product pointwise lane manual and advisory. The P0 calibration behavior is acceptable for local product-intent judging: good fixtures covered the intended items, and bad fixtures remained adverse.