-
Notifications
You must be signed in to change notification settings - Fork 0
test(evals): add technical-design bad pointwise fixtures #31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
13 changes: 13 additions & 0 deletions
13
evals/cases/case-aerial-delivery-shipping-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design uses Shipping for shipping-facing workflow coordination, Delivery for active pickup and delivery execution, Package for package tagging, Scheduler for assignment policy, Accounts for business registration records, and Drone Management for drone availability. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| Delivery owns long-term delivery history. | ||
|
|
||
| This contradicts the source-visible retention boundary: active delivery execution may be separate from long-term delivery history ownership. | ||
13 changes: 13 additions & 0 deletions
13
evals/cases/case-cloudevents-core-contract-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design keeps a CloudEvents Event Contract as the interoperability seam, supports structured and binary JSON encodings, and allows domain-specific payloads to travel between producers and consumers. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| CloudEvents defines event-sourcing lifecycle. | ||
|
|
||
| This invents event-store and event-sourcing ownership that the contract fixture explicitly keeps out of scope. |
13 changes: 13 additions & 0 deletions
13
evals/cases/case-customer-credit-order-saga-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design has an Ordering boundary that creates orders in `PENDING`, sends `reserveCredit` to Customer Credit, and records `APPROVED`, `REJECTED`, `UNKNOWN_CUSTOMER`, and `INSUFFICIENT_CREDIT` outcomes. Messaging and Integration Infrastructure carries commands and asynchronous replies. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| The design uses 2PC for consistency between Order Service and Customer Service. | ||
|
|
||
| This contradicts the local-transaction saga model and the separate service persistence boundary. |
13 changes: 13 additions & 0 deletions
13
evals/cases/case-fineract-loan-lifecycle-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design separates Loan Account Lifecycle, Loan Product Configuration, Loan Charges, and Accounting Integration. Loan Account Lifecycle tracks submitted, approved, active, closed, and overpaid states while reading loan product terms. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| Loan Account Lifecycle owns product configuration. | ||
|
|
||
| This collapses loan product template authority into the account lifecycle boundary. |
13 changes: 13 additions & 0 deletions
13
evals/cases/case-kubernetes-sidecar-containers-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design treats sidecar behavior as part of Pod lifecycle semantics, keeps init-container ordering visible, and discusses Job completion, readiness, resource accounting, and termination ordering. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| Sidecars are ordinary app containers. | ||
|
|
||
| This contradicts the source-visible rule that sidecars are restartable init containers under `initContainers` with `restartPolicy: Always`. |
13 changes: 13 additions & 0 deletions
13
evals/cases/case-openfeature-evaluation-api-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design keeps OpenFeature as a vendor-agnostic evaluation API with providers as backend adapters, no-op default behavior, hook lifecycle stages, and provider readiness/error/configuration/stale events. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| OpenFeature owns flag management UI. | ||
|
|
||
| This invents product-management scope that belongs outside the evaluation API seam. |
13 changes: 13 additions & 0 deletions
13
evals/cases/case-tiny-laundry-pickup-v1/candidate-bad-pointwise.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Expected-Bad Pointwise Candidate | ||
|
|
||
| This candidate is intentionally defective for manual pointwise calibration. It is not a reference design and should remain adverse. | ||
|
|
||
| ## Candidate Design | ||
|
|
||
| The design has Scheduling own booking lifecycle, overlap detection, cancellation rules, appliance availability reads, maintenance hold reads, and the public booking APIs. | ||
|
|
||
| ## Intentional Calibration Defect | ||
|
|
||
| Identity owns scheduling conflict logic. | ||
|
|
||
| This contradicts the source-visible boundary: identity may prove resident verification, but scheduling conflict decisions stay with the booking lifecycle owner. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For these expected-bad pointwise fixtures, this defect section is part of the Markdown that the manual pointwise judge receives as the candidate, so the provider sees both that the file is intentionally bad and what contradiction to look for. That contaminates false-pass calibration: a judge can return red from the answer-key label/explanation instead of detecting a plausible semantic defect in the design itself. Keep this metadata outside the candidate file or strip it before pointwise runs.
Useful? React with 👍 / 👎.