Skip to content

test(evals): add technical-design bad pointwise fixtures#31

Merged
aryeko merged 1 commit into
mainfrom
test/technical-design-bad-pointwise-fixtures
Jul 4, 2026
Merged

test(evals): add technical-design bad pointwise fixtures#31
aryeko merged 1 commit into
mainfrom
test/technical-design-bad-pointwise-fixtures

Conversation

@aryeko

@aryeko aryeko commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add one expected-bad pointwise calibration candidate for each technical-design public case
  • add a deterministic regression test proving each committed bad candidate stays red and hits its targeted contradiction
  • update curated calibration notes to distinguish committed expected-bad candidates from fresh provider evidence

Verification

  • pnpm eval:unit -- --runInBand
  • deterministic pnpm eval:case for all seven candidate-bad-pointwise.md files; each produced verdict red with the expected nonzero exit
  • pnpm install --frozen-lockfile
  • pnpm eval:doctor
  • pnpm eval:list
  • pnpm eval:validate-fixtures
  • pnpm eval:judge:doctor
  • pnpm eval:judge:list
  • pnpm eval:judge:validate-fixtures
  • pnpm check
  • git diff --check

No provider-backed evals were run. Raw c4 evals/results bundles remain ignored/local.

@aryeko aryeko merged commit 9c7bff9 into main Jul 4, 2026
1 check passed
@aryeko aryeko deleted the test/technical-design-bad-pointwise-fixtures branch July 4, 2026 03:40

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ee7c6b721

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +9 to +13
## Intentional Calibration Defect

Delivery owns long-term delivery history.

This contradicts the source-visible retention boundary: active delivery execution may be separate from long-term delivery history ownership.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove answer-key text from judged candidates

For these expected-bad pointwise fixtures, this defect section is part of the Markdown that the manual pointwise judge receives as the candidate, so the provider sees both that the file is intentionally bad and what contradiction to look for. That contaminates false-pass calibration: a judge can return red from the answer-key label/explanation instead of detecting a plausible semantic defect in the design itself. Keep this metadata outside the candidate file or strip it before pointwise runs.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant