Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Treat these as manual/advisory runs, not default CI gates.

```bash
pnpm exec eval-kit judge-coverage \
--config evals/eval-kit.config.json \
--config evals/eval-kit.model-judge.config.json \
--case case-alpha \
--candidate evals/cases/case-alpha/candidate.md \
--model gpt-5.4 \
Expand Down
18 changes: 13 additions & 5 deletions docs/guides/model-assisted-evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,26 +55,34 @@ the consumer has calibrated pointwise evidence and a specific comparison need.

## Commands

Pointwise judging uses the standard manual model-judge config:

```bash
pnpm exec eval-kit generate \
pnpm exec eval-kit judge-coverage \
--config evals/eval-kit.model-judge.config.json \
--case <case-id> \
--candidate <candidate.md> \
--model <model> \
--provider openai \
--effort medium \
--run-id <run-id>
```

Generation and pairwise are separate manual lanes. They require separate configs whose relevant
method is explicitly enabled; do not run them through `evals/eval-kit.config.json` or
`evals/eval-kit.model-judge.config.json`.

pnpm exec eval-kit judge-coverage \
--config evals/eval-kit.model-judge.config.json \
```bash
pnpm exec eval-kit generate \
--config evals/eval-kit.generate.config.json \
--case <case-id> \
--candidate <candidate.md> \
--model <model> \
--provider openai \
--effort medium \
--run-id <run-id>


pnpm exec eval-kit judge-pairwise \
--config evals/eval-kit.pairwise.config.json \
--case <case-id> \
--candidate-a <a.md> \
--candidate-b <b.md> \
Expand Down
10 changes: 7 additions & 3 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,11 +115,13 @@ pnpm exec eval-kit generate \
--provider <openai|openai:codex-app-server> \
--effort <low|medium|high> \
--run-id <id> \
[--config <path>]
--config evals/eval-kit.generate.config.json
```

Requires Promptfoo and local Codex auth. This is manual/advisory evidence, not a default CI gate.
Fails closed before those checks unless `methods.generate.enabled` is explicitly `true`.
Use a separate explicit config for generation; do not enable generation in the deterministic default
config.

## `judge-coverage`

Expand All @@ -133,7 +135,7 @@ pnpm exec eval-kit judge-coverage \
--provider <openai|openai:codex-app-server> \
--effort <low|medium|high> \
[--run-id <id>] \
[--config <path>]
--config evals/eval-kit.model-judge.config.json
```

Fails closed unless `methods.judge_coverage.enabled` is explicitly `true`.
Expand All @@ -157,10 +159,12 @@ pnpm exec eval-kit judge-pairwise \
--effort <low|medium|high> \
--seed <number> \
--run-id <id> \
[--config <path>]
--config evals/eval-kit.pairwise.config.json
```

Fails closed unless `methods.judge_pairwise.enabled` is explicitly `true`.
Use a separate explicit config for pairwise after pointwise calibration; do not enable pairwise in
the deterministic default config.

## `report`

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Current schema:
"run_type": "deterministic",
"runner": {
"id": "generic-eval-case",
"version": "0.1.2"
"version": "0.1.4"
},
"case_ids": ["case-example-v1"],
"started_at": "2026-07-03T00:00:00.000Z",
Expand Down