cli.md

CLI reference

Run:

pnpm exec eval-kit <command> [options]

Package scripts may use plain eval-kit because npm/pnpm add local binaries to script PATH. Interactive consumer shell examples use pnpm exec eval-kit.

Run-producing commands fail closed when their configured method is explicitly disabled:

Command	Disabled by
run-case	methods.deterministic.enabled=false
generate	methods.generate.enabled=false
judge-coverage	methods.judge_coverage.enabled=false
judge-pairwise	methods.judge_pairwise.enabled=false
report	methods.report.enabled=false

Disabled commands exit before Codex auth checks, Promptfoo execution, model provider calls, adapter hooks, or result artifact writes.

Model-assisted commands also require explicit enablement in the selected config before required model/candidate arguments are validated:

Command	Required enablement
generate	methods.generate.enabled=true
judge-coverage	methods.judge_coverage.enabled=true
judge-pairwise	methods.judge_pairwise.enabled=true

init

Create a deterministic generic eval skeleton.

pnpm exec eval-kit init --suite generic [--dry-run] [--force]

Options:

Option	Required	Description
--suite generic	no	Suite template. Defaults to generic.
--dry-run	no	Print planned writes without writing files.
--force	no	Overwrite existing generated files. Use carefully.

scaffold-case

Create a generic deterministic case skeleton.

pnpm exec eval-kit scaffold-case --case <id> [--config <path>] [--dry-run] [--force]

Options:

Option	Required	Description
--case <id>	yes	Path-safe case ID.
--config <path>	no	Config path. Defaults to evals/eval-kit.config.json.
--dry-run	no	Print planned writes without writing files.
--force	no	Overwrite existing case files.

doctor

Validate suite setup.

pnpm exec eval-kit doctor [--config <path>]

list-cases

List discovered case IDs.

pnpm exec eval-kit list-cases [--config <path>]

run-case

Run a deterministic case.

pnpm exec eval-kit run-case \
  --case <id> \
  --candidate <path> \
  [--run-id <id>] \
  [--config <path>]

Exits non-zero when the deterministic verdict is red.

Fails closed when methods.deterministic.enabled is explicitly false.

validate-fixtures

Validate case manifests and call the consumer validateFixtures hook when present.

pnpm exec eval-kit validate-fixtures [--config <path>]

generate

Run Promptfoo-backed candidate generation.

pnpm exec eval-kit generate \
  --case <id> \
  --model <name> \
  --provider <openai|openai:codex-app-server> \
  --effort <low|medium|high> \
  --run-id <id> \
  --config evals/eval-kit.generate.config.json

Requires Promptfoo and local Codex auth. This is manual/advisory evidence, not a default CI gate. Fails closed before those checks unless methods.generate.enabled is explicitly true. Use a separate explicit config for generation; do not enable generation in the deterministic default config.

judge-coverage

Run Promptfoo-backed pointwise coverage judging.

pnpm exec eval-kit judge-coverage \
  --case <id> \
  --candidate <path> \
  --model <name> \
  --provider <openai|openai:codex-app-server> \
  --effort <low|medium|high> \
  [--run-id <id>] \
  --config evals/eval-kit.model-judge.config.json

Fails closed unless methods.judge_coverage.enabled is explicitly true.

judge-pairwise

Run Promptfoo-backed pairwise comparison.

Eval-kit randomizes which original CLI candidate is displayed as Candidate A or Candidate B before calling the consumer adapter. The adapter receives displayed Candidate A/B content and paths plus randomizedOrder, which records the original-to-displayed mapping. Adapters should not apply their own second randomization pass.

pnpm exec eval-kit judge-pairwise \
  --case <id> \
  --candidate-a <path> \
  --candidate-b <path> \
  --model <name> \
  --provider <openai|openai:codex-app-server> \
  --effort <low|medium|high> \
  --seed <number> \
  --run-id <id> \
  --config evals/eval-kit.pairwise.config.json

Fails closed unless methods.judge_pairwise.enabled is explicitly true. Use a separate explicit config for pairwise after pointwise calibration; do not enable pairwise in the deterministic default config.

report

Compile existing run bundles into a unified report through the consumer hook.

pnpm exec eval-kit report \
  --run-id <id> \
  [--generate <id>] \
  [--deterministic <id>] \
  [--judge-coverage <id>] \
  [--judge <id>] \
  [--outcome <id>] \
  [--config <path>]

Fails closed when methods.report.enabled is explicitly false.