diff --git a/evals/model-judge/calibration-notes.md b/evals/model-judge/calibration-notes.md new file mode 100644 index 0000000..1fcc98c --- /dev/null +++ b/evals/model-judge/calibration-notes.md @@ -0,0 +1,41 @@ +# Model-Judge Calibration Notes + +This file is the curated, human-readable calibration summary for the define-product manual pointwise model-judge lane. Raw provider result bundles under `evals/results/**` remain local/ignored and are not committed. + +Deterministic eval verdicts remain authoritative. Pointwise model-judge evidence is manual, advisory, and cannot upgrade deterministic blockers. + +## Run Set + +These notes summarize the existing local provider-backed runs from July 4, 2026. No fresh provider calls were made for this curated summary. + +| Field | Value | +| ----------------- | ---------------------------------------------- | +| Config | `evals/eval-kit.model-judge.config.json` | +| Model | `gpt-5.4` | +| Provider | `openai:codex-app-server` | +| Reasoning effort | `medium` | +| Prompt version | `define-product-pointwise-v1` | +| Rubric version | `define-product-pointwise-rubric-v1` | +| Candidate profile | P0 expected-good and expected-bad PRD fixtures | + +## Calibration Summary + +| Case | Candidate | Run id | Covered | Partial | Missing | Contradicted | Unknown | Human disposition | +| ----------------------------------- | ------------- | ---------------------------------------------------------------- | ------: | ------: | ------: | -----------: | ------: | ----------------------------------------------------------------------------------------------------------- | +| `case-contract-grounded-prd-v1` | expected-good | `provider-20260704-v015b-case-contract-grounded-prd-v1-good` | 5 | 0 | 0 | 0 | 0 | Expected pass-like coverage on product grounding, acceptance criteria, uncertainty, and citation integrity. | +| `case-contract-grounded-prd-v1` | expected-bad | `provider-20260704-v015b-case-contract-grounded-prd-v1-bad` | 0 | 0 | 0 | 5 | 0 | Expected adverse evidence; no pass-like coverage on intended defects. | +| `case-altitude-and-future-scope-v1` | expected-good | `provider-20260704-v015b-case-altitude-and-future-scope-v1-good` | 4 | 0 | 0 | 0 | 0 | Expected pass-like coverage on altitude discipline, future scope, acceptance criteria, and citations. | +| `case-altitude-and-future-scope-v1` | expected-bad | `provider-20260704-v015b-case-altitude-and-future-scope-v1-bad` | 0 | 0 | 0 | 4 | 0 | Expected adverse evidence; no pass-like coverage on intended defects. | + +## Human Calibration Notes + +- False pass risk: no false-pass evidence observed in the P0 expected-bad runs. +- False fail risk: no false-fail evidence observed in the P0 expected-good runs. +- Ambiguity: no `partial` or `unknown` verdicts appeared in these runs. +- Verbosity bias: no evidence that verbose PRD wording was required for coverage. +- Reference wording bias: current evidence is based on local committed fixtures; future expansions should continue to check for semantic coverage rather than reference-text matching. +- Unknown rate: zero unknown verdicts in the summarized run set. + +## Decision + +Keep the define-product pointwise lane manual and advisory. The P0 calibration behavior is acceptable for local product-intent judging: good fixtures covered the intended items, and bad fixtures remained adverse.