feat: add semantic PRD eval suite

evals/cases/README.md

@@ -1,8 +1,20 @@
 # Eval Cases
 
-This directory will hold define-product-owned eval cases in follow-up PRs. Each case owns its
-input material, expected fixture data, rubric, bad-candidate examples, and manifest.
+Each case owns its input material, expected fixture data, rubric, good candidate, bad candidate, and
+manifest. The deterministic adapter grades local Markdown candidates only; model-assisted generation
+and judging remain disabled/manual.
 
-This bootstrap PR intentionally does not add semantic PRD cases. The first case batch should cover
-PRD contract completeness, AC-ID stability, grounding labels, downstream citeability, and product
-altitude.
+## Case Portfolio
+
+- `case-contract-grounded-prd-v1` checks complete PRD contract shape, clear audience/outcome,
+  source-grounded uncertainty labels, non-goals, and downstream citation.
+- `case-revision-preserves-ac-ids-v1` checks stable AC-ID preservation during revision and one new ID
+  for new scope.
+- `case-insufficient-input-blocks-v1` checks fail-closed blocking questions when source material
+  cannot support a coherent PRD.
+- `case-altitude-and-future-scope-v1` checks product altitude, design/runtime leakage rejection, and
+  current-vs-future scope separation.
+
+`pnpm eval:validate-fixtures` validates manifest shape, expected-check schema, and that every
+committed `candidate_good` grades green while every `candidate_bad` grades red. Do not add these
+semantic eval runs to `pnpm check`; they are local, on-demand verification evidence.

evals/cases/case-altitude-and-future-scope-v1/candidate-bad.md

@@ -0,0 +1,34 @@
+# High-Value Refund Review PRD
+
+## Product Outcome
+
+Billing operations gets a React app backed by a Postgres schema and CLI import so Jig execution can
+start after two sprints.
+
+## User Job
+
+As a billing operations reviewer, I need a React queue so engineering can build the right screen.
+
+## Acceptance Criteria
+
+| ID              | Criterion                                                                  | Status |
+| --------------- | -------------------------------------------------------------------------- | ------ |
+| AC-QUEUE-001    | The React app displays high-value refunds from the Postgres schema.        | Active |
+| AC-EVIDENCE-001 | A CLI import loads evidence into the schema.                               | Active |
+| AC-SCOPE-001    | Jig execution begins after two sprints with mobile approvals planned next. | Active |
+
+## Constraints
+
+- Use React, Postgres, a CLI import, and a two-sprint delivery sequence.
+
+## Assumptions
+
+- Inferred: Engineering wants these technologies.
+
+## Non-Goals
+
+- None.
+
+## Downstream Citation Map
+
+Technical Design and Planning may cite `AC-QUEUE-001`, `AC-EVIDENCE-001`, and `AC-SCOPE-001`.

evals/cases/case-altitude-and-future-scope-v1/candidate-good.md

@@ -0,0 +1,48 @@
+# High-Value Refund Review PRD
+
+## Product Outcome
+
+Billing operations reviewers can use a manual refund review queue to decide whether a high-value
+refund is ready for approval.
+
+## User Job
+
+As a billing operations reviewer, I need to see each high-value refund, its stated reason, and
+whether required evidence has been supplied, so I can make an approval-readiness decision.
+
+## Acceptance Criteria
+
+| ID              | Criterion                                                                                                       | Status |
+| --------------- | --------------------------------------------------------------------------------------------------------------- | ------ |
+| AC-QUEUE-001    | Billing operations reviewers can see each high-value refund that needs manual review in one review queue.       | Active |
+| AC-EVIDENCE-001 | Each queued refund shows its reason and whether required evidence has been supplied.                            | Active |
+| AC-SCOPE-001    | The PRD separates current manual-review scope from future mobile approvals, auto-approval rules, and analytics. | Active |
+
+## Constraints
+
+- Success must be recognizable to a billing operations reviewer without reading implementation
+  details.
+- The PRD describes product scope only and does not prescribe how the queue is built or delivered.
+
+## Assumptions
+
+- Inferred: The current owner priority is approval-readiness review, not automated approval.
+- Gap/default: The exact threshold for "high-value refund" is not provided and must be supplied
+  before implementation planning.
+- Gap/default: Technical stakeholder suggestions such as React, Postgres schema, CLI import, two
+  sprints, and Jig execution are design or delivery material rather than product commitments.
+
+## Non-Goals
+
+- No mobile approvals in the current scope.
+- No auto-approval rules in the current scope.
+- No manager analytics integration in the current scope.
+- No commitment to a React implementation, Postgres schema, CLI import, two-sprint delivery plan, or
+  Jig execution mechanics.
+
+## Downstream Citation Map
+
+Technical Design and Planning may cite the Product Outcome, User Job, constraints, assumptions,
+non-goals, and exact IDs `AC-QUEUE-001`, `AC-EVIDENCE-001`, and `AC-SCOPE-001`. They must preserve
+the high-value threshold as a gap/default and must not treat the technical stakeholder suggestions as
+product requirements.

evals/cases/case-altitude-and-future-scope-v1/case-manifest.json

@@ -0,0 +1,16 @@
+{
+  "$schema": "../../../node_modules/@agentic-workflow-kit/eval-kit/schemas/case-manifest.schema.json",
+  "schema_version": "eval-kit.case.v1",
+  "case_id": "case-altitude-and-future-scope-v1",
+  "case_type": "deterministic",
+  "artifacts": [
+    { "role": "generation_visible", "path": "input.md" },
+    { "role": "grader_input", "path": "expected-items.json" },
+    { "role": "rubric", "path": "rubric.md" },
+    { "role": "candidate_good", "path": "candidate-good.md" },
+    { "role": "candidate_bad", "path": "candidate-bad.md" }
+  ],
+  "metadata": {
+    "purpose": "Checks product altitude, design/runtime leakage rejection, and separation of current scope from future work."
+  }
+}

evals/cases/case-altitude-and-future-scope-v1/expected-items.json

@@ -0,0 +1,72 @@
+{
+  "schema_version": "define-product.expected-items.v1",
+  "checks": [
+    {
+      "id": "contract-handoff-valid",
+      "mode": "valid_prd_handoff",
+      "kind": "contract",
+      "severity": "critical"
+    },
+    {
+      "id": "outcome-stays-product-altitude",
+      "mode": "section_includes",
+      "section": "Product Outcome",
+      "kind": "product-altitude",
+      "severity": "high",
+      "must_include_all": [
+        "billing operations reviewers",
+        "manual refund review queue",
+        "ready for approval"
+      ],
+      "must_not_include_any": [
+        "React",
+        "Postgres",
+        "CLI",
+        "schema",
+        "sprint",
+        "Jig"
+      ]
+    },
+    {
+      "id": "expected-ac-ids",
+      "mode": "ac_ids_present",
+      "kind": "citation",
+      "severity": "critical",
+      "ids": ["AC-QUEUE-001", "AC-EVIDENCE-001", "AC-SCOPE-001"]
+    },
+    {
+      "id": "future-work-is-not-current-scope",
+      "mode": "section_includes",
+      "section": "Non-Goals",
+      "kind": "scope",
+      "severity": "critical",
+      "must_include_all": [
+        "mobile approvals",
+        "auto-approval rules",
+        "manager analytics"
+      ]
+    },
+    {
+      "id": "design-leakage-contained",
+      "mode": "forbidden_outside_sections",
+      "kind": "product-altitude",
+      "severity": "critical",
+      "allowed_sections": ["Assumptions", "Non-Goals"],
+      "must_not_include_any": [
+        "React",
+        "Postgres",
+        "CLI",
+        "schema",
+        "two sprints",
+        "Jig execution"
+      ]
+    },
+    {
+      "id": "downstream-map-cites-product-ids",
+      "mode": "downstream_cites_ids",
+      "kind": "citation",
+      "severity": "critical",
+      "ids": ["AC-QUEUE-001", "AC-EVIDENCE-001", "AC-SCOPE-001"]
+    }
+  ]
+}

evals/cases/case-altitude-and-future-scope-v1/input.md

@@ -0,0 +1,18 @@
+# Source Material
+
+Product owner notes:
+
+- Billing operations reviewers need a manual refund review queue so they can decide whether a
+  high-value refund is ready for approval.
+- Current scope: reviewers can see each high-value refund, its reason, and whether required evidence
+  has been supplied.
+- Future work: mobile approvals, auto-approval rules, and integration with manager analytics.
+- A technical stakeholder suggested "React app, Postgres schema, CLI import, two sprints, then Jig
+  execution." Treat that as design/delivery material, not product scope.
+- Success should be recognizable to a reviewer without reading implementation details.
+
+Task:
+
+Draft the product PRD. Keep product sections free of architecture, package layout, CLI behavior,
+schema internals, delivery sequencing, downstream tool mechanics, and runtime details. Separate
+current scope from future work.

evals/cases/case-altitude-and-future-scope-v1/rubric.md

@@ -0,0 +1,10 @@
+# Rubric
+
+This case tests altitude discipline and current/future scope separation.
+
+The candidate should turn the billing-operations notes into product outcomes, user jobs, and
+externally recognizable acceptance criteria. It may mention technical stakeholder suggestions only as
+out-of-scope design/delivery material, not as PRD commitments.
+
+The candidate should fail if architecture, CLI, schema, delivery sequencing, or Jig mechanics appear
+as product requirements or acceptance criteria.

evals/cases/case-contract-grounded-prd-v1/candidate-bad.md

@@ -0,0 +1,33 @@
+# Refund Center PRD
+
+## Product Outcome
+
+Customers can see refunds faster in a new refund center.
+
+## User Job
+
+As a customer, I want my refund resolved.
+
+## Acceptance Criteria
+
+| ID            | Criterion                                           | Status |
+| ------------- | --------------------------------------------------- | ------ |
+| AC-REFUND-001 | The system has refund details.                      | Active |
+| AC-CITE-001   | Teams can read this PRD.                            | Active |
+| AC-GROUND-001 | Legal and operations issues are handled internally. | Active |
+
+## Constraints
+
+- Build a dashboard.
+
+## Assumptions
+
+- Refunds older than 30 days are automatically resolved.
+
+## Non-Goals
+
+- None.
+
+## Downstream Citation Map
+
+- Cite this document.

evals/cases/case-contract-grounded-prd-v1/candidate-good.md

@@ -0,0 +1,48 @@
+# Refund Status Assist PRD
+
+## Product Outcome
+
+Support agents can answer a customer refund-status question in under two minutes from one internal
+view that summarizes refund status and the next owner-visible step.
+
+## User Job
+
+As a support agent, I need to answer a customer refund-status question without searching the order
+tool, payment processor, and support history separately, so I can give the customer a clear status
+and next step during the support interaction.
+
+## Acceptance Criteria
+
+| ID            | Criterion                                                                                                                       | Status |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------ |
+| AC-REFUND-001 | A support agent can identify refund status and the next owner-visible step from one internal view in under two minutes.         | Active |
+| AC-CITE-001   | Technical Design and Planning can cite the refund-status outcome and acceptance-criteria IDs without reading Product internals. | Active |
+| AC-GROUND-001 | The PRD keeps legal wording approval, manual exception uncertainty, and CSV-export disagreement visible as uncertainty.         | Active |
+| AC-SCOPE-001  | The first release excludes automated refund issuance and agent performance analytics.                                           | Active |
+
+## Constraints
+
+- The first release is an internal support-agent experience.
+- Customer-facing status wording must not be treated as settled until Legal approves it.
+- The PRD stays at product altitude and does not prescribe implementation mechanics.
+
+## Assumptions
+
+- Inferred: One internal view is sufficient to replace the three-system lookup for the first release.
+- Gap/default: Refunds older than 30 days may need a manual exception path, but the owner has not
+  settled the product rule yet.
+- Conflict: Manager CSV export was requested by one stakeholder, while the support lead says export
+  is useful later but should not block the first release.
+
+## Non-Goals
+
+- No automated refund issuance.
+- No agent performance analytics.
+- No manager CSV export in the first release unless the owner explicitly changes scope.
+
+## Downstream Citation Map
+
+Technical Design and Planning may cite the Product Outcome, User Job, constraints, assumptions,
+non-goals, and exact IDs `AC-REFUND-001`, `AC-CITE-001`, `AC-GROUND-001`, and `AC-SCOPE-001`.
+They must preserve the legal wording approval, manual exception path, and CSV-export disagreement as
+uncertainty rather than settled product fact.

evals/cases/case-contract-grounded-prd-v1/case-manifest.json

@@ -0,0 +1,16 @@
+{
+  "$schema": "../../../node_modules/@agentic-workflow-kit/eval-kit/schemas/case-manifest.schema.json",
+  "schema_version": "eval-kit.case.v1",
+  "case_id": "case-contract-grounded-prd-v1",
+  "case_type": "deterministic",
+  "artifacts": [
+    { "role": "generation_visible", "path": "input.md" },
+    { "role": "grader_input", "path": "expected-items.json" },
+    { "role": "rubric", "path": "rubric.md" },
+    { "role": "candidate_good", "path": "candidate-good.md" },
+    { "role": "candidate_bad", "path": "candidate-bad.md" }
+  ],
+  "metadata": {
+    "purpose": "Checks a complete, grounded PRD with all contract sections, citable AC IDs, labeled uncertainty, and explicit non-goals."
+  }
+}