Skip to content

feat: add semantic PRD eval suite#8

Merged
aryeko merged 3 commits into
mainfrom
feat/semantic-eval-suite
Jul 3, 2026
Merged

feat: add semantic PRD eval suite#8
aryeko merged 3 commits into
mainfrom
feat/semantic-eval-suite

Conversation

@aryeko

@aryeko aryeko commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the Phase 3 deterministic semantic eval suite for define-product. The suite covers PRD contract completeness, stable AC-ID behavior during revision, insufficient-input blocking behavior, product-altitude boundaries, current/future scope separation, source-grounded uncertainty, and downstream citeability.

Also updates @agentic-workflow-kit/eval-kit to github:agentic-workflow-kit/eval-kit#v0.1.2 so disabled model-assisted commands fail closed in this consumer.

Eval Fixture Behavior

  • Replaces placeholder exact-text grading with deterministic semantic checks over local Markdown candidates.
  • Adds good and bad candidate fixtures for each case.
  • Extends pnpm eval:validate-fixtures to fail closed: it validates the expected-check schema, proves malformed check probes throw, requires good fixtures to grade green, and requires bad fixtures to grade red.
  • Keeps Promptfoo/model-assisted generation and judging disabled/manual; no model-assisted smokes were run.

Verification

  • pnpm install --frozen-lockfile
  • pnpm eval:doctor
  • pnpm eval:list
  • pnpm eval:validate-fixtures
  • pnpm check
  • git diff --check
  • Good deterministic smoke: case-contract-grounded-prd-v1 with candidate-good.md exited 0, verdict green.
  • Bad deterministic smoke: case-contract-grounded-prd-v1 with candidate-bad.md exited 1, verdict red.
  • Malformed expected-check repro now throws before grading: missing text_includes expectations and missing ac_ids_present.ids both fail closed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8b2f76fc8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread evals/adapter.mjs Outdated
Comment thread evals/adapter.mjs Outdated
@aryeko aryeko merged commit 90d9c2d into main Jul 3, 2026
1 check passed
@aryeko aryeko deleted the feat/semantic-eval-suite branch July 3, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant