diff --git a/CHANGELOG.md b/CHANGELOG.md index dd1d45a..2735346 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ documented with migration notes. ## [Unreleased] +### Fixed + +- Marked extraction-era technical-design Promptfoo follow-up docs as historical so they do not + override current two-config pointwise model-judge guidance. +- Replaced a stale release-process dependency example with a version placeholder. + ### Planned - Additional docs for suite-specific adoption. diff --git a/README.md b/README.md index 1e29e03..c3c205f 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,7 @@ v0.1.0 v0.1.1 v0.1.2 v0.1.3 +v0.1.4 v0.1.5 v0.2.0 ``` diff --git a/docs/design/README.md b/docs/design/README.md index 9f5070c..a492553 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -8,5 +8,6 @@ Read: - [`decisions.md`](decisions.md) - append-only decision log. - [`bootstrap-cli.md`](bootstrap-cli.md) - init/scaffold/doctor/list-cases design. - [`consumer-integration.md`](consumer-integration.md) - how consumer repos adopt the package. -- [`technical-design-follow-up.md`](technical-design-follow-up.md) - required technical-design Promptfoo variable fix. +- [`technical-design-follow-up.md`](technical-design-follow-up.md) - historical pre-v0.1.5 + technical-design Promptfoo follow-up; do not use as current integration guidance. - [`skills.md`](skills.md) - agent-facing skill design. diff --git a/docs/design/technical-design-follow-up.md b/docs/design/technical-design-follow-up.md index 556f8f2..9e39b3c 100644 --- a/docs/design/technical-design-follow-up.md +++ b/docs/design/technical-design-follow-up.md @@ -1,64 +1,22 @@ -# Technical-design follow-up +# Historical technical-design follow-up -## Problem +This note records a pre-v0.1.5 follow-up from the original eval-kit extraction. It is not current +integration guidance and should not be used to plan new consumer work. -After extracting eval-kit, `technical-design` consumes the shared package successfully for deterministic evals. The deterministic smoke path passed, but the model-assisted Promptfoo path needs compatibility hardening. +Current model-judge guidance lives in: -The shared generic prompts use generic variables: +- [`../guides/model-assisted-evals.md`](../guides/model-assisted-evals.md) +- [`../guides/model-judge-calibration-reporting.md`](../guides/model-judge-calibration-reporting.md) +- [`../guides/consumer-integration.md`](../guides/consumer-integration.md) -- generation: `source_material`, `candidate_instructions`, `output_format`; -- pointwise: `source_material`, `case_rubric`, `expected_items`, `candidate_path`, `candidate`; -- pairwise: `source_material`, `case_rubric`, `expected_items`, `candidate_a`, `candidate_b`, randomization fields. +## Current policy -Technical-design adapters may still return older variables such as `product_md`, `source_map_md`, `source_facts`, and DDD-specific template keys. That can leave bundled prompts with missing variables. +- Keep `evals/eval-kit.config.json` deterministic and CI-safe. +- Put manual pointwise judge coverage in `evals/eval-kit.model-judge.config.json`. +- Use consumer-owned prompts, expected items, and adapter semantics for the meaning being judged. +- Keep provider-backed commands manual and outside `pnpm check` and CI. +- Treat pairwise and generation lanes as separate, opt-in future work with explicit configs and human + calibration. -## Required fix - -In `technical-design/evals/adapter.mjs`: - -1. Update `resolveGenerationVars` to return: - -```js -{ - source_material: [ - "# Product Brief", - productText, - "# Source Map", - sourceMapText, - ].join("\n\n"), - candidate_instructions: [ - authorSkill, - technicalDesignTemplate, - boundedContextTemplate, - enforcementMapTemplate, - dddEvalExpectations, - ].join("\n\n"), - output_format: "Markdown technical design", -} -``` - -2. Update `resolvePointwiseVars` to return `source_material`. Keep `source_facts` temporarily only if local code still needs it. - -3. Update `resolvePairwiseVars` to return `source_material` and `expected_items` in the shape expected by the shared prompt. - -4. Add tests that call these adapter exports and assert required keys exist. - -5. Run the fast/offline gate and local on-demand deterministic smoke: - -```bash -pnpm check -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit -``` - -Manual/advisory if local auth exists: - -```bash -pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model --provider openai --effort medium --run-id verify-generation-vars -pnpm eval:judge:coverage -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --model --provider openai --effort medium --run-id verify-pointwise-vars -``` - -## Alternative - -If technical-design should keep fully specialized prompts, add local prompt templates under `technical-design/evals/prompts/` and configure `prompt_templates` in `evals/eval-kit.config.json`. - -Preferred path: align the adapter to generic variable names and keep specialized details inside `candidate_instructions` and source material. +The old extraction-era advice to wire generation and pairwise variables for `technical-design` is no +longer required by the standard pointwise model-judge lane. diff --git a/docs/guides/technical-design-integration-fix.md b/docs/guides/technical-design-integration-fix.md index 9f979d0..a6a084c 100644 --- a/docs/guides/technical-design-integration-fix.md +++ b/docs/guides/technical-design-integration-fix.md @@ -1,67 +1,20 @@ -# Fixing the technical-design integration +# Historical technical-design integration fix -This guide is for the follow-up PR in `agentic-workflow-kit/technical-design`. +This guide is retained as extraction-era history only. It described a `technical-design` follow-up for +early eval-kit tags and should not be used as current guidance. -## Current issue +For current consumer setup, use: -`technical-design` consumes `@agentic-workflow-kit/eval-kit` through `github:agentic-workflow-kit/eval-kit#v0.1.0`. Deterministic evals pass, but model-assisted Promptfoo commands need variable alignment with the shared generic prompts. +- [`consumer-integration.md`](consumer-integration.md) +- [`model-assisted-evals.md`](model-assisted-evals.md) +- [`model-judge-calibration-reporting.md`](model-judge-calibration-reporting.md) -## Fix steps +Current consumers should keep the two-config pattern: -1. Read `technical-design/AGENTS.md` and `technical-design/evals/README.md`. -2. Inspect `evals/eval-kit.config.json`. -3. Inspect `evals/adapter.mjs`. -4. Update generation, pointwise, and pairwise variable resolvers to include the generic keys expected by eval-kit prompts. -5. Add unit tests for adapter variable contracts. -6. Run the fast/offline `pnpm check` gate and the local on-demand deterministic smoke. +- `evals/eval-kit.config.json` remains deterministic and CI-safe. +- `evals/eval-kit.model-judge.config.json` owns manual pointwise judge coverage. +- `eval:judge:*` scripts target the model-judge config. +- Provider-backed commands remain manual and outside `pnpm check` and CI. -## Required keys - -Generation: - -```text -source_material -candidate_instructions -output_format -``` - -Pointwise: - -```text -source_material -case_rubric -expected_items -candidate_path -candidate -``` - -Pairwise: - -```text -source_material -case_rubric -expected_items -candidate_a -candidate_b -randomization_method -randomization_seed -original_order -candidate_order -``` - -## Verification - -Required: - -```bash -pnpm install --frozen-lockfile -pnpm check -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-shared-eval-kit -``` - -Manual/advisory if auth is available: - -```bash -pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model --provider openai --effort medium --run-id verify-generation-vars -pnpm eval:judge:coverage -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --model --provider openai --effort medium --run-id verify-pointwise-vars -``` +Do not revive generation or pairwise wiring from this historical note as required work. Generation +and pairwise lanes require separate explicit configs, consumer-owned semantics, and human calibration. diff --git a/docs/reference/release-process.md b/docs/reference/release-process.md index 9e56429..23638fc 100644 --- a/docs/reference/release-process.md +++ b/docs/reference/release-process.md @@ -6,7 +6,7 @@ Consumers depend on tags like: ```json { - "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.0" + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#vX.Y.Z" } ```