Skip to content

feat: add define-product model judge lane#9

Merged
aryeko merged 2 commits into
mainfrom
phase-5a-model-judge
Jul 3, 2026
Merged

feat: add define-product model judge lane#9
aryeko merged 2 commits into
mainfrom
phase-5a-model-judge

Conversation

@aryeko

@aryeko aryeko commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the Phase 5A manual/advisory pointwise model-judge foundation for define-product. The deterministic eval suite remains the authoritative gate; this adds a separate model-judge config, PRD pointwise prompt, local expected-item schema, P0 pointwise items for the grounded contract and altitude/future-scope cases, and adapter hooks for pointwise vars and metadata canonicalization.

The manual judge lane is kept outside pnpm check and CI. Promptfoo is added as a local dev dependency so the enabled model-judge config passes doctor, but no provider calls were run.

Verification

  • pnpm format
  • pnpm eval:doctor
  • ./node_modules/.bin/eval-kit doctor --config evals/eval-kit.model-judge.config.json
  • pnpm eval:list
  • pnpm eval:validate-fixtures
  • ./node_modules/.bin/eval-kit validate-fixtures --config evals/eval-kit.model-judge.config.json
  • deterministic smoke: both P0 good candidates pass and both bad candidates fail red
  • pnpm check
  • git diff --check

No provider/model calls were run.

@aryeko

aryeko commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ccc11698a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread package.json
Comment thread evals/eval-kit.model-judge.config.json Outdated
@aryeko aryeko merged commit 572b103 into main Jul 3, 2026
1 check passed
@aryeko aryeko deleted the phase-5a-model-judge branch July 3, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant