Skip to content

chore(evals): standardize model judge config#24

Merged
aryeko merged 1 commit into
mainfrom
standard-model-judge-configs
Jul 3, 2026
Merged

chore(evals): standardize model judge config#24
aryeko merged 1 commit into
mainfrom
standard-model-judge-configs

Conversation

@aryeko

@aryeko aryeko commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add the standard manual pointwise model-judge config alongside the deterministic default config
  • add standard eval:judge:* helper scripts and pin eval-kit to v0.1.3
  • update eval docs so users validate/run pointwise judging through evals/eval-kit.model-judge.config.json instead of editing the default config

Verification

  • pnpm install
  • pnpm format
  • pnpm eval:doctor
  • pnpm eval:list
  • pnpm eval:validate-fixtures
  • pnpm eval:judge:doctor
  • pnpm eval:judge:list
  • pnpm eval:judge:validate-fixtures
  • pnpm check
  • git diff --check

Did not run eval:judge:coverage or provider/model calls.

@aryeko

aryeko commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@aryeko aryeko merged commit c762a66 into main Jul 3, 2026
1 check passed
@aryeko aryeko deleted the standard-model-judge-configs branch July 3, 2026 23:49

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca33058465

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/README.md

Use any account-supported model approved for the local calibration run. Examples below use
`gpt-5.5`; that is an example, not repo policy. The standard manual config judges an existing
candidate. If a future workflow needs provider-backed candidate generation, add a separate explicit

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Omit the nonexistent generate run from reports

When following this new no-generation flow, the user is told to judge an existing candidate and not enable generation, but the report command below still passes --generate <run-id>-generate; compileReport then unconditionally reads that supplied generate run's report.md (evals/adapter.mjs:355-358), so the documented flow fails unless the user happened to create an unrelated generation bundle. Please either omit --generate from this manual pointwise-only report command or add a matching generation step/config.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant