From 4105a424116801385e2dd811b2c4afaad73b5140 Mon Sep 17 00:00:00 2001 From: Arye Kogan Date: Sat, 4 Jul 2026 06:30:47 +0300 Subject: [PATCH] docs(evals): add model-judge calibration notes --- evals/model-judge-calibration-notes.md | 44 ++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 evals/model-judge-calibration-notes.md diff --git a/evals/model-judge-calibration-notes.md b/evals/model-judge-calibration-notes.md new file mode 100644 index 0000000..a43f378 --- /dev/null +++ b/evals/model-judge-calibration-notes.md @@ -0,0 +1,44 @@ +# Model-Judge Calibration Notes + +This file is the curated, human-readable calibration summary for the technical-design manual pointwise model-judge lane. Raw provider result bundles under `evals/results/**` remain local/ignored and are not committed. + +Deterministic eval verdicts remain authoritative. Pointwise model-judge evidence is manual, advisory, and cannot upgrade deterministic blockers. + +## Run Set + +These notes summarize the existing local provider-backed runs from July 4, 2026. No fresh provider calls were made for this curated summary. + +| Field | Value | +| ----------------- | -------------------------------------------- | +| Config | `evals/eval-kit.model-judge.config.json` | +| Model | `gpt-5.4` | +| Provider | `openai:codex-app-server` | +| Reasoning effort | `medium` | +| Prompt version | `pointwise-prompt-v1` | +| Rubric version | `pointwise-coverage-rubric-v1` | +| Candidate profile | expected-good `reference-design.md` fixtures | + +## Calibration Summary + +| Case | Candidate | Run id | Covered | Partial | Missing | Contradicted | Unknown | Human disposition | +| --------------------------------------- | ------------- | ------------------------------------------------------------------------ | ------: | ------: | ------: | -----------: | ------: | ---------------------------------------------------- | +| `case-aerial-delivery-shipping-v1` | expected-good | `provider-20260704-v015-case-aerial-delivery-shipping-v1-reference` | 16 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-cloudevents-core-contract-v1` | expected-good | `provider-20260704-v015-case-cloudevents-core-contract-v1-reference` | 6 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-customer-credit-order-saga-v1` | expected-good | `provider-20260704-v015-case-customer-credit-order-saga-v1-reference` | 11 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-fineract-loan-lifecycle-v1` | expected-good | `provider-20260704-v015-case-fineract-loan-lifecycle-v1-reference` | 10 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-kubernetes-sidecar-containers-v1` | expected-good | `provider-20260704-v015-case-kubernetes-sidecar-containers-v1-reference` | 9 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-openfeature-evaluation-api-v1` | expected-good | `provider-20260704-v015-case-openfeature-evaluation-api-v1-reference` | 8 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | +| `case-tiny-laundry-pickup-v1` | expected-good | `provider-20260704-v015-case-tiny-laundry-pickup-v1-reference` | 12 | 0 | 0 | 0 | 0 | Pass-like coverage; no calibration concern observed. | + +## Human Calibration Notes + +- False pass risk: not yet calibrated for technical-design because this run set contains expected-good pointwise candidates only. Phase C4 is the planned follow-up for expected-bad pointwise calibration fixtures. +- False fail risk: no false-fail evidence observed in the expected-good run set. +- Ambiguity: no `partial` or `unknown` verdicts appeared in these runs. +- Verbosity bias: no evidence that longer wording was required for coverage, but expected-bad fixtures are needed before drawing stronger conclusions. +- Reference wording bias: current evidence only proves that the judge recognizes committed reference designs; it does not yet prove rejection of plausible but semantically defective designs. +- Unknown rate: zero unknown verdicts in the summarized run set. + +## Decision + +Keep the technical-design pointwise lane manual and advisory. The current expected-good evidence is usable for smoke calibration, but false-pass detection remains untested until Phase C4 adds expected-bad pointwise calibration fixtures.