diff --git a/docs/design/promptfoo-codex-eval-pilot.md b/docs/design/promptfoo-codex-eval-pilot.md index 600d61e..aabc742 100644 --- a/docs/design/promptfoo-codex-eval-pilot.md +++ b/docs/design/promptfoo-codex-eval-pilot.md @@ -116,24 +116,20 @@ This design follows the docs for these reasons: The implemented boundary separates generic mechanics from the consuming suite: -```text -packages/eval-kit/ # generic CLI, schemas, Promptfoo helpers, bundled prompts -evals/ # technical-design cases, adapter, rubrics, tests, results -``` - -Generic package name: - ```text @agentic-workflow-kit/eval-kit +evals/ # technical-design cases, adapter, rubrics, tests, results ``` -This package is internal infrastructure for the `technical-design` repo. It must stay private and -must not be treated as part of the product surface shipped to users. Root `pnpm eval:*` scripts are -the consumer command surface and call the `eval-kit` binary with `evals/eval-kit.config.json`. +`@agentic-workflow-kit/eval-kit` is shared infrastructure consumed from the GitHub tag pinned in +`package.json` and `pnpm-lock.yaml`. It must stay private and must not be treated as part of the +product surface shipped to users. Root `pnpm eval:*` scripts are the consumer command surface and +call the installed `eval-kit` binary with `evals/eval-kit.config.json`. Eval-specific material is split by ownership: -- generic runner code, portable schemas, and bundled Promptfoo prompts -> `packages/eval-kit/`; +- generic runner code, portable schemas, and bundled Promptfoo prompts -> the shared + `@agentic-workflow-kit/eval-kit` package; - `evals/README.md` and `evals/implementation-plan.md` -> consumer docs. - `evals/tests/**` -> `evals/tests/`. - `evals/schemas/**` -> `evals/schemas/`. @@ -147,7 +143,7 @@ Eval-specific material is split by ownership: Root-level docs may still describe the evaluation strategy, because that strategy is part of how the product is designed and governed. Executable eval assets, fixture data, generated outputs, schemas, -and eval-specific implementation docs belong in the internal package. +and eval-specific implementation docs belong in `evals/`. The root package stays small. It keeps `check` as the public deterministic repo gate and exposes manual eval commands that call the internal kit: @@ -176,7 +172,7 @@ Add `promptfoo` as a development dependency of the eval package and add manual s Keep `pnpm check` deterministic-only. -The Promptfoo templates live in `packages/eval-kit/promptfoo/` and suite-specific variables come +The Promptfoo templates live in the shared `@agentic-workflow-kit/eval-kit` package and suite-specific variables come from `evals/adapter.mjs`: - Generation suite: @@ -196,7 +192,7 @@ from `evals/adapter.mjs`: - Provider: `openai:codex-app-server`. - Model: `gpt-5.4`. - Reasoning effort: medium. - - Output schema: `packages/eval-kit/schemas/pairwise-result.schema.json`. + - Output schema: `@agentic-workflow-kit/eval-kit/schemas/pairwise-result.schema.json`. - Test variables: case id, source facts, expected facts and boundaries, generated candidate, reference anchor, candidate order, original order, randomization method, and seed. - Assertions: JSON output, schema-valid output, evidence present, model/provider/rubric/prompt @@ -206,7 +202,7 @@ from `evals/adapter.mjs`: - Provider: `openai:codex-app-server`. - Model: `gpt-5.4`. - Reasoning effort: medium. - - Output schema: `packages/eval-kit/schemas/pointwise-judge-result.schema.json`. + - Output schema: `@agentic-workflow-kit/eval-kit/schemas/pointwise-judge-result.schema.json`. - Test variables: case id, source facts, expected facts and boundaries, generated candidate, and deterministic grades if available. - Assertions: JSON output, schema-valid output, evidence on every covered/partial/contradicted @@ -252,7 +248,8 @@ The final report command combines all run bundles and states: - Promptfoo, not shell wrappers, owns provider execution, assertions, and JSON/HTML result exports. - Candidate generation produces a non-empty Markdown design for `case-tiny-laundry-pickup-v1`. - The existing deterministic grader runs against the generated candidate. -- The judge output validates against `packages/eval-kit/schemas/pairwise-result.schema.json`. +- The judge output validates against + `@agentic-workflow-kit/eval-kit/schemas/pairwise-result.schema.json`. - The final report is written under ignored `evals/results/**`. - Generated result files are not committed unless a human reviewer explicitly promotes a redacted summary. diff --git a/evals/README.md b/evals/README.md index 45071a7..0729965 100644 --- a/evals/README.md +++ b/evals/README.md @@ -2,9 +2,9 @@ This directory contains the consumer-facing eval suite for the `technical-design` skills pack. -It uses [`@agentic-workflow-kit/eval-kit`](../packages/eval-kit) for all generic mechanics -(schema validation, path resolution, Promptfoo helpers, result manifests). This directory owns -only the `technical-design` domain: DDD fixtures, expected facts and boundaries, judge rubrics, +It uses `@agentic-workflow-kit/eval-kit` from the shared GitHub tag for all generic mechanics +(schema validation, path resolution, Promptfoo helpers, result manifests). This directory owns only +the `technical-design` domain: DDD fixtures, expected facts and boundaries, judge rubrics, lessons-ledger checks, and deterministic grading policy. The layered design-quality strategy lives in diff --git a/evals/adapter.mjs b/evals/adapter.mjs index a666689..db6d63f 100644 --- a/evals/adapter.mjs +++ b/evals/adapter.mjs @@ -711,11 +711,11 @@ export const validateFixtures = async ({ config, manifests }) => { ]; const expectedFactsShape = compileLocalSchema( - "packages/eval-kit/schemas/expected-facts.schema.json", + "evals/schemas/expected-facts.schema.json", "expected facts", ); const expectedBoundariesShape = compileLocalSchema( - "packages/eval-kit/schemas/expected-boundaries.schema.json", + "evals/schemas/expected-boundaries.schema.json", "expected boundaries", ); @@ -910,6 +910,8 @@ export const gradeTechnicalDesignCandidate = ({ }; }; +export const gradeCandidate = gradeTechnicalDesignCandidate; + export const criticalBlockerCount = (findings) => countPolicyBlockers(findings, deterministicVerdictPolicy); diff --git a/evals/eval-kit.config.json b/evals/eval-kit.config.json index 1a08c3e..42199ab 100644 --- a/evals/eval-kit.config.json +++ b/evals/eval-kit.config.json @@ -1,5 +1,5 @@ { - "$schema": "../packages/eval-kit/schemas/eval-kit.config.schema.json", + "$schema": "../node_modules/@agentic-workflow-kit/eval-kit/schemas/eval-kit.config.schema.json", "schema_version": "eval-kit.config.v1", "suite_id": "technical-design", "suite_root": ".", diff --git a/evals/implementation-plan.md b/evals/implementation-plan.md index 05f9d6e..b02b57d 100644 --- a/evals/implementation-plan.md +++ b/evals/implementation-plan.md @@ -275,11 +275,11 @@ Recommended OSS tools: Implemented: -- Added the suite judge rubric at `evals/rubric.md`; bundled generic judge prompts live under - `packages/eval-kit/promptfoo/judges/`. +- Added the suite judge rubric at `evals/rubric.md`; bundled generic judge prompts live in the + shared `@agentic-workflow-kit/eval-kit` package. - Added a pairwise comparison prompt that requires randomized candidate order to be recorded. -- Added bundled schemas for structured judge output and pairwise results under - `packages/eval-kit/schemas/`. +- Added bundled schemas for structured judge output and pairwise results in the shared + `@agentic-workflow-kit/eval-kit` package. - Added a manual Promptfoo template outside the required PR gate. Remaining: diff --git a/packages/eval-kit/schemas/expected-boundaries.schema.json b/evals/schemas/expected-boundaries.schema.json similarity index 97% rename from packages/eval-kit/schemas/expected-boundaries.schema.json rename to evals/schemas/expected-boundaries.schema.json index 79e3c8f..c4ad3ef 100644 --- a/packages/eval-kit/schemas/expected-boundaries.schema.json +++ b/evals/schemas/expected-boundaries.schema.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/expected-boundaries.schema.json", + "$id": "https://agentic-workflow-kit.local/technical-design/evals/expected-boundaries.schema.json", "title": "ExpectedBoundaries", "type": "object", "required": ["case_id", "contexts"], diff --git a/packages/eval-kit/schemas/expected-facts.schema.json b/evals/schemas/expected-facts.schema.json similarity index 97% rename from packages/eval-kit/schemas/expected-facts.schema.json rename to evals/schemas/expected-facts.schema.json index afeaebe..a3baf87 100644 --- a/packages/eval-kit/schemas/expected-facts.schema.json +++ b/evals/schemas/expected-facts.schema.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/expected-facts.schema.json", + "$id": "https://agentic-workflow-kit.local/technical-design/evals/expected-facts.schema.json", "title": "ExpectedFacts", "type": "object", "required": ["case_id", "facts"], diff --git a/evals/tests/validate-eval-fixtures.test.mjs b/evals/tests/validate-eval-fixtures.test.mjs index 164dd1e..e126e18 100644 --- a/evals/tests/validate-eval-fixtures.test.mjs +++ b/evals/tests/validate-eval-fixtures.test.mjs @@ -55,13 +55,7 @@ const createFixtureRepo = () => { fs.symlinkSync( path.join(repoRoot, "node_modules"), - path.join(tempDir, "evals/node_modules"), - ); - - fs.mkdirSync(path.join(tempDir, "packages"), { recursive: true }); - fs.symlinkSync( - path.resolve(repoRoot, "packages/eval-kit"), - path.join(tempDir, "packages/eval-kit"), + path.join(tempDir, "node_modules"), ); return tempDir; @@ -70,14 +64,23 @@ const createFixtureRepo = () => { const runValidator = (cwd) => { const evalKitBin = path.resolve( repoRoot, - "packages/eval-kit/bin/eval-kit.mjs", + "node_modules/@agentic-workflow-kit/eval-kit/bin/eval-kit.mjs", ); try { - execFileSync(process.execPath, [evalKitBin, "validate-fixtures"], { - cwd: path.join(cwd, "evals"), - encoding: "utf8", - stdio: "pipe", - }); + execFileSync( + process.execPath, + [ + evalKitBin, + "validate-fixtures", + "--config", + path.join(cwd, "evals/eval-kit.config.json"), + ], + { + cwd: path.join(cwd, "evals"), + encoding: "utf8", + stdio: "pipe", + }, + ); return { ok: true, stdout: "", stderr: "" }; } catch (error) { return { diff --git a/package.json b/package.json index 51b2804..6cf6432 100644 --- a/package.json +++ b/package.json @@ -33,7 +33,7 @@ "check:links": "node scripts/check_non_eval.mjs links", "check:non-eval": "node scripts/check_non_eval.mjs all", "smoke:enforce-generator": "node scripts/smoke_generate_depcruise.mjs", - "check": "pnpm lint && pnpm check:non-eval && pnpm smoke:enforce-generator && pnpm --filter @agentic-workflow-kit/eval-kit test && pnpm check:deterministic", + "check": "pnpm lint && pnpm check:non-eval && pnpm smoke:enforce-generator && pnpm check:deterministic", "check:deterministic": "pnpm check:eval-static && pnpm eval:unit && pnpm eval:enforce", "check:eval-static": "node scripts/check_eval_static.mjs", "eval:unit": "vitest run evals/tests", @@ -45,7 +45,7 @@ "eval:manual-report": "eval-kit report --config evals/eval-kit.config.json" }, "devDependencies": { - "@agentic-workflow-kit/eval-kit": "workspace:*", + "@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.0", "ajv": "^8.17.1", "dependency-cruiser": "^18.0.0", "prettier": "^3.6.2", diff --git a/packages/eval-kit/README.md b/packages/eval-kit/README.md deleted file mode 100644 index 35847f9..0000000 --- a/packages/eval-kit/README.md +++ /dev/null @@ -1,469 +0,0 @@ -# Eval Kit - -`@agentic-workflow-kit/eval-kit` is a portable runner package for local evaluation suites. It is -currently a private workspace package inside `technical-design`, but it is written as a standalone -kit that can later move to a shared repository. - -The kit owns reusable mechanics: - -- command routing for deterministic and model-assisted eval runs; -- config loading and path containment; -- case manifest discovery; -- JSON Schema registration and validation; -- result manifest and artifact records; -- Promptfoo execution helpers and bundled prompt templates; -- generic text coverage helpers and verdict aggregation. - -The consuming repository owns domain semantics: - -- eval cases and fixture files; -- graders and reporters; -- Promptfoo variable resolution adapter exports; -- domain schemas and fixture validation; -- rubrics and human interpretation. - -## Status - -Current supported surfaces: - -- `run-case` deterministic grading through consumer-provided grader and reporter modules; -- `validate-fixtures` manifest validation plus optional consumer fixture validation; -- `generate` Promptfoo generation using the bundled generation prompt or a consumer override; -- `judge-coverage` pointwise Promptfoo judging using the bundled pointwise prompt; -- `report` combined manual report assembly through a consumer hook; -- result manifests using `eval-kit.result-manifest.v2`; -- schema and path helper APIs exported from `src/index.mjs`. - -Known implementation follow-ups before publishing this as a fully generic shared package: - -- consumer-shaped schemas should be reviewed and either generalized or moved to consumers; -- pairwise judging is gated by `methods.judge_pairwise.enabled`; -- provider support is intentionally narrow: `openai` and `openai:codex-app-server`; -- model-assisted commands require local Codex auth and Promptfoo installed in the consumer repo. - -## Package Layout - -```text -packages/eval-kit/ - bin/eval-kit.mjs # executable entry point - src/ - cli.mjs # command parser and command dispatch - config.mjs # config loading, schema roots, hook loading - paths.mjs # safe path/id helpers - schema.mjs # Ajv 2020 schema registry - artifacts.mjs # artifact records and result manifests - promptfoo.mjs # Promptfoo execution and output parsing - grading.mjs # generic normalized text coverage helpers - verdict.mjs # generic verdict aggregation - sdk.mjs # command implementations - index.mjs # public exports - schemas/ # bundled schemas - promptfoo/ - generation.prompt.md # default generation prompt - judges/ - pointwise.prompt.md # default pointwise judge prompt - pairwise.prompt.md # default pairwise judge prompt - tests/ # package unit tests -``` - -Companion docs: - -- [Adapter Contract](./docs/adapter-contract.md) -- [Schemas](./docs/schemas.md) -- [Examples](./docs/examples.md) - -## Core Design - -Eval Kit has two boundaries. - -The kit boundary is generic infrastructure. It can load a suite, resolve a case, execute a command, -validate outputs, and write a result bundle. It must not import consumer content directly. - -The suite boundary is domain-specific interpretation. A suite decides what a case means, what -artifacts are visible to a model, how a candidate is graded, and how reports should read. - -At runtime, the kit loads `eval-kit.config.json`, resolves the suite root and result root, registers -the bundled schemas plus suite schema roots, then dynamically imports suite modules declared in the -config. - -```mermaid -flowchart LR - CLI["eval-kit CLI"] - Config["eval-kit.config.json"] - Kit["Eval Kit runtime"] - Suite["Consumer adapter"] - Cases["Case manifests and fixtures"] - Results["Result bundle"] - - CLI --> Config - Config --> Kit - Kit --> Cases - Kit --> Suite - Suite --> Kit - Kit --> Results -``` - -## Installation - -Today, the package is consumed as a workspace dependency: - -```json -{ - "devDependencies": { - "@agentic-workflow-kit/eval-kit": "workspace:*" - } -} -``` - -The package exposes: - -```json -{ - "bin": { - "eval-kit": "./bin/eval-kit.mjs" - }, - "exports": { - ".": "./src/index.mjs" - } -} -``` - -For model-assisted commands, the consuming repository also needs: - -- `promptfoo` available at `node_modules/.bin/promptfoo`; -- local Codex auth available through `codex login status`; -- the suite adapter exports required by the chosen command. - -## Minimal Consumer Layout - -```text -evals/ - eval-kit.config.json - adapter.mjs - cases/ - case-example-v1/ - case-manifest.json - product.md - expected-facts.json - expected-boundaries.json - rubric.md - reference-design.md - results/ - README.md -``` - -The paths above are conventions, not requirements. The config decides the suite root, result root, -case discovery, method settings, and adapter module. - -## Configuration - -A typical consumer config: - -```json -{ - "$schema": "../packages/eval-kit/schemas/eval-kit.config.schema.json", - "schema_version": "eval-kit.config.v1", - "suite_id": "technical-design", - "suite_root": ".", - "results_root": "results", - "adapter": "adapter.mjs", - "cases": { - "root": "cases", - "include": ["case-*-v1"] - }, - "methods": { - "deterministic": { - "enabled": true, - "grader": "facts-boundaries", - "reporter": "markdown" - }, - "generate": { - "enabled": true, - "prompt": "@eval-kit/generation" - }, - "judge_coverage": { - "enabled": true, - "prompt": "@eval-kit/pointwise", - "rubric": "case:rubric.md" - }, - "judge_pairwise": { - "enabled": false - }, - "report": { - "enabled": true - } - } -} -``` - -Path behavior: - -- the config file path is resolved from the current working directory; -- `suite_root`, `results_root`, and relative adapter paths resolve from the config directory; -- `cases.root` resolves from `suite_root`; -- `cases.include` and `cases.exclude` match immediate case directory names with `*` wildcards; -- suite and result paths must stay contained by the detected repository root; -- run ids and case ids are treated as ids, not paths. - -Prompt behavior: - -- `prompt_templates.generation` overrides the bundled generation prompt; -- `prompt_templates.pointwise_judge` overrides the bundled pointwise prompt; -- `prompt_templates.pairwise_judge` overrides the bundled pairwise prompt. - -If a prompt template is omitted, the kit uses its bundled prompt. - -Legacy configs that declare `case_manifests`, `graders`, `reporters`, `hooks`, and `schema_roots` -are still accepted for compatibility, but new suites should prefer `adapter`, `cases`, and -`methods`. - -## Case Manifests - -Each case manifest must include `schema_version`, `case_id`, and `artifacts`. Additional -consumer-specific fields are allowed. - -```json -{ - "schema_version": "technical-design.case-manifest.v1", - "case_id": "case-tiny-laundry-pickup-v1", - "case_type": "tiny_contract", - "artifacts": [ - { "role": "generation_visible", "path": "product.md" }, - { "role": "grader_input", "path": "expected-facts.json" }, - { "role": "grader_input", "path": "expected-boundaries.json" }, - { "role": "semantic_reference", "path": "reference-design.md" } - ], - "grading": { - "grader": "technical-design-facts-boundaries" - } -} -``` - -Artifact paths are relative to the manifest directory and must exist. The kit adds -`absolutePath` when returning resolved artifacts to adapter and SDK functions. - -## CLI - -The binary is `eval-kit`. - -In the current `technical-design` consumer, root scripts wrap the binary: - -```bash -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md -pnpm eval:generate -- --case case-tiny-laundry-pickup-v1 --model gpt-5.4 --provider openai --effort medium --run-id tiny-generate -pnpm eval:judge:coverage -- --case case-tiny-laundry-pickup-v1 --candidate evals/results/tiny-generate/cases/case-tiny-laundry-pickup-v1/candidate.md --model gpt-5.4 --provider openai --effort medium -pnpm eval:manual-report -- --run-id tiny-report --deterministic tiny-deterministic --judge-coverage tiny-pointwise -``` - -Direct binary usage: - -```bash -eval-kit run-case \ - --config evals/eval-kit.config.json \ - --case case-tiny-laundry-pickup-v1 \ - --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md \ - --run-id verify-tiny-reference -``` - -### Commands - -`run-case` - -Deterministically grades one candidate file. It loads the configured grader and reporter, writes a -result bundle, and exits non-zero when the verdict is `red`. - -Required: - -- `--case ` -- `--candidate ` - -Optional: - -- `--run-id ` -- `--config ` - -`generate` - -Runs Promptfoo through the Codex App Server provider and writes a generated candidate. - -Required: - -- `--case ` -- `--model ` -- `--provider ` -- `--effort ` -- `--run-id ` - -Optional: - -- `--config ` - -`judge-coverage` - -Runs a pointwise model judge over expected items and a candidate. - -Required: - -- `--case ` -- `--candidate ` -- `--model ` -- `--provider ` -- `--effort ` - -Optional: - -- `--run-id ` -- `--config ` - -`judge-pairwise` - -Compares two candidates with randomized order. The command fails closed when -`methods.judge_pairwise.enabled` is explicitly `false`. - -`report` - -Combines existing run bundles into one manual report through the consumer `compileReport` hook. - -Required: - -- `--run-id ` - -Optional parent runs: - -- `--generate ` -- `--deterministic ` -- `--judge-coverage ` -- `--judge ` -- `--outcome ` -- `--config ` - -`validate-fixtures` - -Validates discovered case manifests and then calls `adapter.validateFixtures`, if present. - -## Adapter Contract - -The kit loads the suite adapter from config. A single `adapter.mjs` can implement all roles. Legacy -configs may still split graders and reporters into separate modules. - -Deterministic grading expects a grader export named one of: - -- `gradeTechnicalDesignCandidate` -- `gradeCandidate` -- `default` - -The grader receives: - -```js -{ - candidateText, - ...graderInputs -} -``` - -`graderInputs` are built from artifacts with role `grader_input`; JSON file names become camelCase -keys. For example, `expected-facts.json` becomes `expectedFacts`. - -The grader returns: - -```js -{ - verdict: "green", - findings: [] -} -``` - -Reporting expects a reporter export named one of: - -- `renderDeterministicReport` -- `renderReport` -- `default` - -Promptfoo and manual-report commands expect named hook exports. See -[Adapter Contract](./docs/adapter-contract.md) for the complete hook signatures. - -## Result Bundles - -Every command writes into: - -```text -// - manifest.json - report.md - ...command-specific artifacts -``` - -The current v2 manifest shape is: - -```json -{ - "schema_version": "eval-kit.result-manifest.v2", - "run_id": "verify-tiny-reference", - "run_type": "deterministic", - "runner": { - "id": "technical-design-eval-case", - "version": "0.0.0" - }, - "case_ids": ["case-tiny-laundry-pickup-v1"], - "started_at": "2026-07-01T00:00:00.000Z", - "ended_at": "2026-07-01T00:00:01.000Z", - "duration_ms": 1000, - "status": "completed", - "git": { - "commit": "abc123" - }, - "command": "eval-kit run-case ...", - "tool_versions": { - "node": "v26.0.0" - }, - "artifacts": [], - "output_files": ["manifest.json", "report.md"] -} -``` - -Artifacts include path, media type, byte size, SHA-256, encoding, and redaction status. - -## Public API - -The package exports these helpers from `@agentic-workflow-kit/eval-kit`: - -- args: `defaultRunId`, `parseArgs`, `requireArg` -- paths: `assertContainedPath`, `assertSafeId`, `containsPath`, `createPathResolver`, - `toPosixPath` -- schemas: `createSchemaRegistry`, `validateWithSchema` -- artifacts: `artifactRecord`, `normalizeLegacyManifest`, `sha256File`, `writeManifest`, - `writeResultBundle` -- Promptfoo: `extractPromptfooOutput`, `parseJsonOutput`, `runPromptfooRaw` -- verdicts: `aggregateVerdict`, `criticalBlockerCount` -- config: `loadConfig` -- grading helpers: `normalize`, `candidateSegments`, `includesAny`, `includesAll`, - `assessCoverage`, `gradeFacts` -- SDK commands: `resolveCaseManifest`, `discoverCaseIds`, `runCase`, `generateCandidate`, - `judgeCoverage`, `judgePairwise`, `compileReport`, `validateFixtures` - -## Verification - -Package tests: - -```bash -pnpm --filter @agentic-workflow-kit/eval-kit test -``` - -In this repository, the full gate is: - -```bash -pnpm check -``` - -Manual deterministic smoke: - -```bash -pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-tiny-reference -``` - -Expected report headline: - -```text -Verdict: green -Blocker findings: 0 -``` diff --git a/packages/eval-kit/bin/eval-kit.mjs b/packages/eval-kit/bin/eval-kit.mjs deleted file mode 100755 index 965a925..0000000 --- a/packages/eval-kit/bin/eval-kit.mjs +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env node -import { main } from "../src/cli.mjs"; -main(); diff --git a/packages/eval-kit/docs/adapter-contract.md b/packages/eval-kit/docs/adapter-contract.md deleted file mode 100644 index aaec9b5..0000000 --- a/packages/eval-kit/docs/adapter-contract.md +++ /dev/null @@ -1,301 +0,0 @@ -# Eval Kit Adapter Contract - -Eval Kit stays generic by loading a consumer adapter dynamically. The consumer provides domain -behavior through exports from the adapter declared in `eval-kit.config.json`. - -## Loading Rules - -Adapter module paths are resolved from `suite_root`. - -```json -{ - "suite_root": ".", - "adapter": "adapter.mjs", - "methods": { - "deterministic": { - "enabled": true, - "grader": "facts-boundaries", - "reporter": "markdown" - } - } -} -``` - -The adapter module can implement all exports. Legacy configs that split graders, reporters, and -hooks into separate module declarations remain supported. - -## Deterministic Grader - -The `run-case` command picks: - -- `methods.deterministic.grader`, when set; otherwise -- the legacy `runner_defaults.grader`, when set; otherwise -- the first key in legacy `graders`. - -The module must export one of: - -```js -export const gradeTechnicalDesignCandidate = ({ - candidateText, - expectedFacts, -}) => { - return { - verdict: "green", - findings: [], - }; -}; - -export const gradeCandidate = ({ candidateText }) => ({ - verdict: "green", - findings: [], -}); - -export default ({ candidateText }) => ({ verdict: "green", findings: [] }); -``` - -Input object: - -```js -{ - candidateText: "...", - expectedFacts: {...}, - expectedBoundaries: {...} -} -``` - -The kit builds extra keys from manifest artifacts with role `grader_input`: - -- read JSON from each artifact; -- remove `.json`; -- camel-case dash-separated names. - -Examples: - -- `expected-facts.json` becomes `expectedFacts`; -- `expected-boundaries.json` becomes `expectedBoundaries`. - -Return object: - -```js -{ - verdict: "red" | "yellow" | "green" | "great", - findings: [ - { - id: "FACT-001", - kind: "fact", - severity: "critical", - verdict: "covered", - evidence: "exact evidence hit" - } - ] -} -``` - -The return value is validated with `grades.schema.json`. - -## Deterministic Reporter - -The `run-case` command picks: - -- `methods.deterministic.reporter`, when set; otherwise -- the legacy `runner_defaults.reporter`, when set; otherwise -- the first key in legacy `reporters`. - -The module must export one of: - -```js -export const renderDeterministicReport = (input) => "# Eval Report\n"; -export const renderReport = (input) => "# Eval Report\n"; -export default (input) => "# Eval Report\n"; -``` - -Input object: - -```js -{ - (caseId, grades, findings, caseDir, candidatePath, resolver); -} -``` - -Return a Markdown string. The kit writes it to `report.md`. - -## Generation Hook - -The `generate` command requires: - -```js -export const resolveGenerationVars = async ({ - caseId, - caseDir, - artifacts, - resolver, -}) => { - return { - product_md: "...", - source_map_md: "...", - author_skill_md: "...", - technical_design_template_md: "...", - bounded_context_template_md: "...", - enforcement_map_template_md: "...", - ddd_eval_expectations_md: "...", - }; -}; -``` - -The returned object becomes Promptfoo `vars`. The bundled generation prompt currently references -the variable names shown above. Consumers can either return those variables or override -`prompt_templates.generation` in config. - -## Pointwise Judge Hook - -The `judge-coverage` command requires: - -```js -export const resolvePointwiseVars = async ({ - caseId, - caseDir, - artifacts, - candidateContent, - candidatePath, - promptVersion, - rubricVersion, - model, - provider, - resolver -}) => { - return { - case_id: caseId, - model, - provider, - prompt_version: promptVersion, - rubric_version: rubricVersion, - source_facts: "...", - expected_items: JSON.stringify([...], null, 2), - candidate_path: resolver.relativeToRepo(candidatePath), - candidate: candidateContent, - _expectedItemsForCanonicalization: [...] - }; -}; -``` - -Optional post-processing: - -```js -export const canonicalizeExpectedItemMetadata = ( - actualItems, - expectedItems, -) => { - return expectedItems.map((expected) => ({ - ...actualItems.find((item) => item.item_id === expected.item_id), - kind: expected.kind, - severity: expected.severity, - source_refs: expected.source_refs, - })); -}; -``` - -Use this hook when the model should judge content but not rewrite trusted metadata. - -## Pairwise Judge Hook - -The `judge-pairwise` command requires: - -```js -export const resolvePairwiseVars = async ({ - caseId, - caseDir, - artifacts, - candidateAContent, - candidateBContent, - candidateAPath, - candidateBPath, - promptVersion, - rubricVersion, - model, - provider, - randomizedOrder, - resolver, -}) => { - return { - case_id: caseId, - model, - provider, - prompt_version: promptVersion, - rubric_version: rubricVersion, - source_facts: "...", - case_rubric: "...", - expected_facts: "...", - candidate_a: candidateAContent, - candidate_b: candidateBContent, - randomization_method: randomizedOrder.method, - randomization_seed: randomizedOrder.seed, - original_order: randomizedOrder.original_order.join(", "), - candidate_order: randomizedOrder.candidate_order.join(", "), - }; -}; -``` - -Pairwise execution is controlled by the consumer config. If -`methods.judge_pairwise.enabled` is explicitly `false`, the CLI fails closed before running -Promptfoo. - -## Manual Report Hook - -The `report` command requires: - -```js -export const compileReport = async ({ - config, - runId, - runs, - resultDir, - resolver, -}) => { - return { - reportContent: "# Manual Combined Report\n", - caseIds: ["case-example-v1"], - artifacts: [ - { - role: "deterministic_grades", - path: "deterministic_grades.json", - mediaType: "application/json", - }, - ], - outputFiles: ["deterministic_grades.json"], - }; -}; -``` - -The hook is responsible for copying or writing any extra artifacts it lists. Eval Kit always writes -`report.md` and `manifest.json`. - -## Fixture Validation Hook - -The `validate-fixtures` command validates configured case manifests, then calls: - -```js -export const validateFixtures = async ({ config, manifests }) => { - for (const { manifest, fullPath, relativePath } of manifests) { - // suite-specific checks - } -}; -``` - -Throw an error to fail validation. - -## Resolver API - -Hooks receive `resolver`, created by `createPathResolver`. - -Useful methods: - -- `resolveRunDir(runId)` -- `resolveResultArtifact(runDir, relativePath, label)` -- `resolveSuitePath(relativePath, label)` -- `resolveRepoPath(relativePath, label)` -- `relativeToRepo(absolutePath)` -- `relativeToSuite(absolutePath)` -- `relativeToResults(absolutePath)` - -All path methods enforce containment. Run ids use `assertSafeId`, so ids cannot contain path -separators. diff --git a/packages/eval-kit/docs/examples.md b/packages/eval-kit/docs/examples.md deleted file mode 100644 index d07147f..0000000 --- a/packages/eval-kit/docs/examples.md +++ /dev/null @@ -1,219 +0,0 @@ -# Eval Kit Examples - -These examples use the current `technical-design` consumer layout, but the same patterns apply to -any suite that provides config, manifests, and adapters. - -## Deterministic Reference Case - -Run the tiny reference design through deterministic grading: - -```bash -pnpm eval:case -- \ - --case case-tiny-laundry-pickup-v1 \ - --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md \ - --run-id verify-tiny-reference -``` - -Expected files: - -```text -evals/results/verify-tiny-reference/ - manifest.json - grades.json - report.md - cases/case-tiny-laundry-pickup-v1/candidate.md - cases/case-tiny-laundry-pickup-v1/grader-output.json -``` - -Expected report headline: - -```text -Verdict: green -Blocker findings: 0 -``` - -## Direct Binary Invocation - -```bash -eval-kit run-case \ - --config evals/eval-kit.config.json \ - --case case-tiny-laundry-pickup-v1 \ - --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md \ - --run-id direct-tiny-reference -``` - -## Minimal Grader - -```js -export const gradeCandidate = ({ candidateText, expectedFacts }) => { - const findings = expectedFacts.facts.map((fact) => ({ - id: fact.id, - kind: "fact", - severity: fact.severity, - verdict: candidateText.includes(fact.must_include_any[0]) - ? "covered" - : "missing", - evidence: candidateText.includes(fact.must_include_any[0]) - ? `found ${fact.must_include_any[0]}` - : `missing ${fact.must_include_any[0]}`, - })); - - const hasBlocker = findings.some( - (finding) => - finding.severity === "critical" && finding.verdict !== "covered", - ); - - return { - verdict: hasBlocker ? "red" : "green", - findings, - }; -}; -``` - -## Minimal Reporter - -```js -export const renderReport = ({ caseId, grades, findings }) => { - return [ - `# Eval Report: ${caseId}`, - "", - `Verdict: ${grades.verdict}`, - "", - "## Findings", - "", - ...findings.map( - (finding) => - `- ${finding.id} (${finding.severity}): ${finding.verdict} - ${finding.evidence}`, - ), - ].join("\n"); -}; -``` - -## Fixture Validation - -```bash -eval-kit validate-fixtures --config evals/eval-kit.config.json -``` - -The kit validates discovered case manifests first. If `adapter.validateFixtures` exists, the kit -calls it with: - -```js -{ - config, - manifests: [ - { - manifest, - fullPath, - relativePath - } - ] -} -``` - -## Candidate Generation - -```bash -pnpm eval:generate -- \ - --case case-tiny-laundry-pickup-v1 \ - --model gpt-5.4 \ - --provider openai \ - --effort medium \ - --run-id tiny-generate -``` - -This writes: - -```text -evals/results/tiny-generate/ - manifest.json - report.md - promptfooconfig.json - promptfoo-results.json - promptfoo-report.html - cases/case-tiny-laundry-pickup-v1/candidate.md -``` - -Requirements: - -- `promptfoo` installed in the consumer repo; -- `codex login status` succeeds; -- `adapter.resolveGenerationVars` returns variables expected by the prompt. - -## Pointwise Judge - -```bash -pnpm eval:judge:coverage -- \ - --case case-tiny-laundry-pickup-v1 \ - --candidate evals/results/tiny-generate/cases/case-tiny-laundry-pickup-v1/candidate.md \ - --model gpt-5.4 \ - --provider openai \ - --effort medium \ - --run-id tiny-pointwise -``` - -This writes: - -```text -evals/results/tiny-pointwise/ - manifest.json - report.md - pointwise-result.json - promptfooconfig.json - promptfoo-results.json - promptfoo-report.html -``` - -## Combined Manual Report - -```bash -pnpm eval:manual-report -- \ - --run-id tiny-report \ - --deterministic verify-tiny-reference \ - --judge-coverage tiny-pointwise -``` - -The suite hook decides how to combine parent runs. Eval Kit writes a v2 result manifest for the -combined report. - -## Programmatic Use - -```js -import { - loadConfig, - runCase, - discoverCaseIds, -} from "@agentic-workflow-kit/eval-kit"; - -const config = loadConfig("evals/eval-kit.config.json"); -const cases = discoverCaseIds(config); - -await runCase({ - config, - caseId: cases[0], - candidatePath: "evals/cases/case-tiny-laundry-pickup-v1/reference-design.md", - runId: "programmatic-run", -}); -``` - -## Safe Paths - -```js -import { createPathResolver } from "@agentic-workflow-kit/eval-kit"; - -const resolver = createPathResolver({ - repoRoot: process.cwd(), - configDir: `${process.cwd()}/evals`, - suiteRoot: ".", - resultsRoot: "results", -}); - -const runDir = resolver.resolveRunDir("example-run"); -const manifestPath = resolver.resolveResultArtifact( - runDir, - "manifest.json", - "manifest", -); -``` - -`resolveRunDir("../bad")` throws because run ids must be ids, not paths. diff --git a/packages/eval-kit/docs/schemas.md b/packages/eval-kit/docs/schemas.md deleted file mode 100644 index 4bc100d..0000000 --- a/packages/eval-kit/docs/schemas.md +++ /dev/null @@ -1,249 +0,0 @@ -# Eval Kit Schemas - -Bundled schemas live in `packages/eval-kit/schemas/`. They are registered automatically by -`loadConfig`, then any optional consumer `schema_roots` are added. - -The schema registry is strict: - -- duplicate `$id` values fail config loading; -- schemas can be validated by file name, such as `grades.schema.json`; -- schemas can be validated by `$id`; -- validation errors include the instance path and message. - -## Portable Schemas - -These are the core generic contracts. - -### `eval-kit.config.schema.json` - -Validates the suite config. - -Required fields: - -- `schema_version`: must be `eval-kit.config.v1`; -- `suite_id`: stable suite id used in runner ids; -- `suite_root`: path to suite files, relative to config file; -- `results_root`: path for run outputs, relative to config file. - -Optional fields: - -- `adapter`: suite adapter module path, relative to `suite_root`; -- `cases.root`: case directory root, relative to `suite_root`; -- `cases.include`: immediate case directory patterns to include; -- `cases.exclude`: immediate case directory patterns to exclude; -- `methods`: deterministic, generation, judge, and report method settings; -- `prompt_templates`: generation, pointwise, and pairwise prompt overrides; -- `schema_roots`: extra schema directories, relative to config file. - -Legacy compatibility fields remain accepted: - -- `case_manifests`: explicit manifest paths, relative to `suite_root`; -- `artifact_roles`: suite-defined role names; -- `runner_defaults`: runner-specific defaults; -- `graders`: map of grader name to module path; -- `reporters`: map of reporter name to module path; -- `hooks`: module path for generation, judge, report, and validation hooks. - -### `case-manifest.schema.json` - -Validates manifest-level structure. - -Required fields: - -- `schema_version` -- `case_id` -- `artifacts` - -Each artifact requires: - -- `role` -- `path` - -The schema allows additional manifest fields so consumers can add grading, grounding, provenance, -or case taxonomy metadata. - -### `artifact.schema.json` - -Validates result artifact records. - -Required fields: - -- `role` -- `path` -- `sha256` -- `size_bytes` -- `media_type` -- `redaction_status` - -Allowed `redaction_status` values: - -- `raw-local` -- `redacted` -- `public-safe` -- `legacy-unknown` - -### `result-manifest.v2.schema.json` - -The current result bundle manifest. - -Required fields: - -- `schema_version`: must be `eval-kit.result-manifest.v2`; -- `run_id`; -- `run_type`; -- `runner.id` and `runner.version`; -- `case_ids`; -- `started_at`; -- `ended_at`; -- `duration_ms`; -- `status`: `completed` or `runner_failed`; -- `git.commit`; -- `command`; -- `tool_versions`; -- `artifacts`; -- `output_files`. - -Optional model-run fields: - -- `model` -- `provider` -- `model_provider` -- `reasoning_effort` -- `sandbox_mode` -- `approval_policy` -- `codex_auth_mode` -- `prompt_version` -- `rubric_version` -- `randomization` -- `provenance.parent_run_ids` - -### `finding.schema.json` - -Generic minimal finding shape: - -- `id` -- `severity` -- `verdict` -- `evidence` - -Additional properties are allowed. - -### `run-report.schema.json` - -Small summary contract: - -- `run_id` -- `status` -- `summary` - -## Current Consumer-Shaped Schemas - -These schemas live in eval-kit today because they were part of the extraction from -`technical-design`. They should be reviewed before publishing eval-kit as a fully generic shared -package. - -### `grades.schema.json` - -Current deterministic result shape: - -- `case_id` -- `verdict`: `red`, `yellow`, `green`, or `great`; -- `findings[]`. - -Each finding currently requires: - -- `id` -- `kind`: `fact` or `boundary`; -- `severity`: `critical` or `recommended`; -- `verdict`: `covered`, `missing`, `contradicted`, `invented`, or `unknown`; -- `evidence`. - -### `expected-facts.schema.json` - -Technical-design expected fact fixture shape. It supports: - -- required source refs; -- critical or recommended severity; -- include-any and include-all text expectations; -- forbidden snippets; -- accepted alternatives; -- required concept groups. - -### `expected-boundaries.schema.json` - -Technical-design expected boundary fixture shape. It supports: - -- context ids and names; -- owned, read, and not-owned responsibilities; -- required and forbidden snippets; -- accepted alternatives; -- required concept groups. - -### `pointwise-judge-result.schema.json` - -Pointwise model-judge result. The judge returns one item per expected fact or boundary: - -- `item_id` -- `kind` -- `verdict`: `covered`, `partial`, `missing`, `contradicted`, or `unknown`; -- `severity` -- `confidence` -- `candidate_evidence` -- `source_refs` -- `explanation` - -When the verdict is `covered`, `partial`, or `contradicted`, candidate evidence must be non-empty. - -### `judge-output.schema.json` - -Legacy single-criterion judge output shape retained for compatibility. It requires: - -- `case_id` -- `criterion` -- `verdict`: `pass`, `fail`, or `unknown`; -- `severity` -- `evidence` -- `explanation` -- `confidence` -- `model` -- `rubric_version` - -### `pairwise-result.schema.json` - -Pairwise judge output and stored result schema. It requires: - -- `case_id` -- `model` -- `provider` -- `rubric_version` -- `prompt_version` -- `candidate_order` -- `randomization` -- `winner` -- `criteria` -- `evidence` -- `explanation` -- `confidence` - -### `results-manifest.schema.json` - -Legacy manifest schema retained for compatibility. New output should use -`result-manifest.v2.schema.json`. - -## Schema Registry API - -```js -import { createSchemaRegistry } from "@agentic-workflow-kit/eval-kit"; - -const registry = createSchemaRegistry({ - schemaRoots: ["packages/eval-kit/schemas", "evals/schemas"], -}); - -const manifest = registry.validateWithSchema( - "result-manifest.v2.schema.json", - data, - "result manifest", -); -``` - -The registry returns the input data when valid and throws on validation failure. diff --git a/packages/eval-kit/package.json b/packages/eval-kit/package.json deleted file mode 100644 index f0ef29e..0000000 --- a/packages/eval-kit/package.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "name": "@agentic-workflow-kit/eval-kit", - "version": "0.0.0", - "description": "Portable eval runner primitives for local eval suites.", - "private": true, - "type": "module", - "exports": { - ".": "./src/index.mjs" - }, - "bin": { - "eval-kit": "./bin/eval-kit.mjs" - }, - "scripts": { - "test": "vitest run tests" - }, - "license": "MIT", - "dependencies": { - "ajv": "^8.17.1" - }, - "devDependencies": { - "vitest": "^3.2.4" - } -} diff --git a/packages/eval-kit/promptfoo/generation.prompt.md b/packages/eval-kit/promptfoo/generation.prompt.md deleted file mode 100644 index a62db72..0000000 --- a/packages/eval-kit/promptfoo/generation.prompt.md +++ /dev/null @@ -1,23 +0,0 @@ -# Candidate Generation Prompt - -Prompt version: `generation-prompt-v1`. - -Generate one technical design candidate for the requested eval case. - -Inputs: - -- Case id: `{{case_id}}` -- Product brief: `{{product_md}}` -- Source map: `{{source_map_md}}` -- Author skill: `{{author_skill_md}}` -- Technical design template: `{{technical_design_template_md}}` -- Bounded context template: `{{bounded_context_template_md}}` -- Enforcement map template: `{{enforcement_map_template_md}}` -- Eval expectations: `{{ddd_eval_expectations_md}}` - -Use only the visible product and source inputs. Do not infer requirements from hidden reference -designs, grader notes, or prior runs. Preserve explicit product goals, constraints, non-goals, and -source-visible ownership boundaries. When evidence is missing, name the uncertainty instead of -inventing implementation facts. - -Return Markdown for the candidate technical design only. diff --git a/packages/eval-kit/promptfoo/judges/pairwise.prompt.md b/packages/eval-kit/promptfoo/judges/pairwise.prompt.md deleted file mode 100644 index cc41f7c..0000000 --- a/packages/eval-kit/promptfoo/judges/pairwise.prompt.md +++ /dev/null @@ -1,53 +0,0 @@ -# Pairwise Regression Prompt - -Prompt version: `pairwise-prompt-v1`. - -Compare two candidate technical designs for the same case. Candidate order is randomized and must be -recorded in the result metadata with method, seed, original order, and candidate order. - -Inputs: - -- Case id: `{{case_id}}` -- Model: `{{model}}` -- Provider: `{{provider}}` -- Prompt version: `{{prompt_version}}` -- Rubric version: `{{rubric_version}}` -- Product/source facts: `{{source_facts}}` -- Case rubric: `{{case_rubric}}` -- Expected facts and boundaries: `{{expected_facts}}` -- Candidate A: `{{candidate_a}}` -- Candidate B: `{{candidate_b}}` -- Randomization method: `{{randomization_method}}` -- Randomization seed: `{{randomization_seed}}` -- Original candidate order: `{{original_order}}` -- Candidate order: `{{candidate_order}}` - -## Rubric - -Rubric version: `judge-rubric-v1`. - -Use this rubric only after deterministic graders have passed and any pointwise coverage judge results -have been reviewed. Pairwise comparison chooses the stronger candidate overall; it does not prove -that every expected fact or boundary is covered. - -| Criterion | Severity | Pass signal | -| ------------------- | ----------- | ---------------------------------------------------------------------------------- | -| Source preservation | critical | Candidate preserves product goals, non-goals, constraints, and required workflows. | -| Boundary coherence | critical | Context ownership is explicit and does not blur producer-owned decisions. | -| Invariant clarity | critical | Guarded predicates name operands, authority, and failure behavior. | -| Planning usefulness | recommended | Delivery facts, sequencing, validation, and stop conditions are extractable. | -| Ceremony fit | recommended | DDD depth fits the case complexity without adding unsupported tactical ceremony. | - -## Bias Controls - -- Return `unknown` when evidence is insufficient. -- Cite candidate and expected-fact evidence in every result. -- Do not infer missing facts from reference design wording. -- Do not reward length, familiar architecture vocabulary, or rhetorical confidence without source - support. -- Do not override deterministic or pointwise coverage blockers. If a candidate is better overall but - still misses required evidence, explain that limitation. - -Choose which candidate is more source-grounded, implementation-ready, and enforceable. - -Return JSON matching `packages/eval-kit/schemas/pairwise-result.schema.json`. diff --git a/packages/eval-kit/promptfoo/judges/pointwise.prompt.md b/packages/eval-kit/promptfoo/judges/pointwise.prompt.md deleted file mode 100644 index 7f61abc..0000000 --- a/packages/eval-kit/promptfoo/judges/pointwise.prompt.md +++ /dev/null @@ -1,46 +0,0 @@ -# Pointwise Coverage Prompt - -Prompt version: `pointwise-prompt-v1`. - -Grade one candidate technical design item-by-item against the visible source inputs and expected -coverage items for the case. This is an advisory coverage pass, not a holistic design ranking. - -Inputs: - -- Case id: `{{case_id}}` -- Model: `{{model}}` -- Provider: `{{provider}}` -- Prompt version: `{{prompt_version}}` -- Rubric version: `{{rubric_version}}` -- Product/source inputs: `{{source_facts}}` -- Case rubric: `{{case_rubric}}` -- Expected items: `{{expected_items}}` -- Candidate path: `{{candidate_path}}` -- Candidate: `{{candidate}}` - -## Rubric - -Rubric version: `pointwise-coverage-rubric-v1`. - -For each expected item: - -- Use `covered` only when the candidate clearly preserves the item with direct supporting excerpts. -- Use `partial` when the candidate addresses part of the item but omits a required detail. -- Use `missing` when the candidate does not provide enough support for the item. -- Use `contradicted` when the candidate conflicts with the expected item. -- Use `unknown` only when the candidate is too ambiguous to judge even after reading the visible - sources and expected item metadata. - -## Bias Controls - -- Judge only against `product.md`, `source-map.md`, expected facts, expected boundaries, and the - candidate. Use `rubric.md` only for semantic guidance, not hidden deterministic requirements. -- Do not use or infer from `reference-design.md`, grader notes, review rubrics, or hidden answer - keys. -- Do not reward length, familiar architecture vocabulary, or rhetorical confidence without source - support. -- `covered`, `partial`, and `contradicted` must each include at least one direct candidate evidence - excerpt. -- Preserve each item's `item_id`, `kind`, `severity`, and `source_refs` exactly as provided. - -Return JSON matching `packages/eval-kit/schemas/pointwise-judge-result.schema.json`. diff --git a/packages/eval-kit/schemas/artifact.schema.json b/packages/eval-kit/schemas/artifact.schema.json deleted file mode 100644 index 7323c6c..0000000 --- a/packages/eval-kit/schemas/artifact.schema.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/artifact.schema.json", - "title": "EvalKitArtifact", - "type": "object", - "additionalProperties": false, - "required": [ - "role", - "path", - "sha256", - "size_bytes", - "media_type", - "redaction_status" - ], - "properties": { - "role": { - "type": "string", - "minLength": 1 - }, - "path": { - "type": "string", - "minLength": 1 - }, - "sha256": { - "anyOf": [ - { - "type": "string", - "pattern": "^[a-f0-9]{64}$" - }, - { - "type": "null" - } - ] - }, - "size_bytes": { - "type": "integer", - "minimum": 0 - }, - "media_type": { - "type": "string", - "minLength": 1 - }, - "encoding": { - "type": "string", - "minLength": 1 - }, - "redaction_status": { - "enum": ["raw-local", "redacted", "public-safe", "legacy-unknown"] - } - } -} diff --git a/packages/eval-kit/schemas/case-manifest.schema.json b/packages/eval-kit/schemas/case-manifest.schema.json deleted file mode 100644 index 3c69ce8..0000000 --- a/packages/eval-kit/schemas/case-manifest.schema.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/case-manifest.schema.json", - "title": "EvalKitCaseManifest", - "type": "object", - "additionalProperties": true, - "required": ["schema_version", "case_id", "artifacts"], - "properties": { - "schema_version": { - "type": "string", - "minLength": 1 - }, - "case_id": { - "type": "string", - "minLength": 1 - }, - "case_type": { - "type": "string" - }, - "artifacts": { - "type": "array", - "minItems": 1, - "items": { - "type": "object", - "additionalProperties": false, - "required": ["role", "path"], - "properties": { - "role": { - "type": "string", - "minLength": 1 - }, - "path": { - "type": "string", - "minLength": 1 - } - } - } - }, - "metadata": { - "type": "object" - } - } -} diff --git a/packages/eval-kit/schemas/eval-kit.config.schema.json b/packages/eval-kit/schemas/eval-kit.config.schema.json deleted file mode 100644 index 77d0db2..0000000 --- a/packages/eval-kit/schemas/eval-kit.config.schema.json +++ /dev/null @@ -1,153 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/eval-kit.config.schema.json", - "title": "EvalKitConfig", - "type": "object", - "additionalProperties": false, - "required": ["schema_version", "suite_id", "suite_root", "results_root"], - "properties": { - "$schema": { - "type": "string" - }, - "schema_version": { - "const": "eval-kit.config.v1" - }, - "suite_id": { - "type": "string", - "minLength": 1 - }, - "suite_root": { - "type": "string", - "minLength": 1 - }, - "results_root": { - "type": "string", - "minLength": 1 - }, - "schema_roots": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - }, - "case_manifests": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "adapter": { - "type": "string", - "minLength": 1 - }, - "cases": { - "type": "object", - "additionalProperties": false, - "required": ["root"], - "properties": { - "root": { "type": "string", "minLength": 1 }, - "include": { - "type": "array", - "minItems": 1, - "items": { "type": "string", "minLength": 1 } - }, - "exclude": { - "type": "array", - "items": { "type": "string", "minLength": 1 } - } - } - }, - "artifact_roles": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "runner_defaults": { - "type": "object" - }, - "graders": { - "type": "object", - "additionalProperties": { - "type": "string", - "minLength": 1 - } - }, - "reporters": { - "type": "object", - "additionalProperties": { - "type": "string", - "minLength": 1 - } - }, - "prompt_templates": { - "type": "object", - "additionalProperties": false, - "properties": { - "generation": { "type": "string", "minLength": 1 }, - "pointwise_judge": { "type": "string", "minLength": 1 }, - "pairwise_judge": { "type": "string", "minLength": 1 } - } - }, - "hooks": { - "type": "object", - "additionalProperties": false, - "required": ["module"], - "properties": { - "module": { "type": "string", "minLength": 1 } - } - }, - "methods": { - "type": "object", - "additionalProperties": false, - "properties": { - "deterministic": { - "type": "object", - "additionalProperties": false, - "properties": { - "enabled": { "type": "boolean" }, - "grader": { "type": "string", "minLength": 1 }, - "reporter": { "type": "string", "minLength": 1 } - } - }, - "generate": { - "type": "object", - "additionalProperties": false, - "properties": { - "enabled": { "type": "boolean" }, - "prompt": { "type": "string", "minLength": 1 } - } - }, - "judge_coverage": { - "type": "object", - "additionalProperties": false, - "properties": { - "enabled": { "type": "boolean" }, - "prompt": { "type": "string", "minLength": 1 }, - "rubric": { "type": "string", "minLength": 1 } - } - }, - "judge_pairwise": { - "type": "object", - "additionalProperties": false, - "properties": { - "enabled": { "type": "boolean" }, - "prompt": { "type": "string", "minLength": 1 }, - "rubric": { "type": "string", "minLength": 1 } - } - }, - "report": { - "type": "object", - "additionalProperties": false, - "properties": { - "enabled": { "type": "boolean" } - } - } - } - } - } -} diff --git a/packages/eval-kit/schemas/finding.schema.json b/packages/eval-kit/schemas/finding.schema.json deleted file mode 100644 index ecd519c..0000000 --- a/packages/eval-kit/schemas/finding.schema.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/finding.schema.json", - "title": "EvalKitFinding", - "type": "object", - "additionalProperties": true, - "required": ["id", "severity", "verdict", "evidence"], - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "severity": { - "type": "string", - "minLength": 1 - }, - "verdict": { - "type": "string", - "minLength": 1 - }, - "evidence": { - "type": "string", - "minLength": 1 - } - } -} diff --git a/packages/eval-kit/schemas/grades.schema.json b/packages/eval-kit/schemas/grades.schema.json deleted file mode 100644 index 12eb8b2..0000000 --- a/packages/eval-kit/schemas/grades.schema.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/grades.schema.json", - "title": "CaseGrades", - "type": "object", - "required": ["case_id", "verdict", "findings"], - "additionalProperties": false, - "properties": { - "case_id": { - "type": "string", - "minLength": 1 - }, - "verdict": { - "enum": ["red", "yellow", "green", "great"] - }, - "findings": { - "type": "array", - "items": { - "type": "object", - "required": ["id", "kind", "severity", "verdict", "evidence"], - "additionalProperties": false, - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "kind": { - "enum": ["fact", "boundary"] - }, - "severity": { - "enum": ["critical", "recommended"] - }, - "verdict": { - "enum": [ - "covered", - "missing", - "contradicted", - "invented", - "unknown" - ] - }, - "evidence": { - "type": "string", - "minLength": 1 - } - } - } - } - } -} diff --git a/packages/eval-kit/schemas/judge-output.schema.json b/packages/eval-kit/schemas/judge-output.schema.json deleted file mode 100644 index 64fd26f..0000000 --- a/packages/eval-kit/schemas/judge-output.schema.json +++ /dev/null @@ -1,57 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/judge-output.schema.json", - "title": "JudgeOutput", - "type": "object", - "required": [ - "case_id", - "criterion", - "verdict", - "severity", - "evidence", - "explanation", - "confidence", - "model", - "rubric_version" - ], - "additionalProperties": false, - "properties": { - "case_id": { - "type": "string", - "minLength": 1 - }, - "criterion": { - "type": "string", - "minLength": 1 - }, - "verdict": { - "enum": ["pass", "fail", "unknown"] - }, - "severity": { - "enum": ["critical", "recommended"] - }, - "evidence": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - }, - "explanation": { - "type": "string", - "minLength": 1 - }, - "confidence": { - "enum": ["low", "medium", "high"] - }, - "model": { - "type": "string", - "minLength": 1 - }, - "rubric_version": { - "type": "string", - "minLength": 1 - } - } -} diff --git a/packages/eval-kit/schemas/pairwise-result.schema.json b/packages/eval-kit/schemas/pairwise-result.schema.json deleted file mode 100644 index fbee256..0000000 --- a/packages/eval-kit/schemas/pairwise-result.schema.json +++ /dev/null @@ -1,118 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/pairwise-result.schema.json", - "title": "PairwiseJudgeResult", - "type": "object", - "required": [ - "case_id", - "model", - "provider", - "rubric_version", - "prompt_version", - "candidate_order", - "randomization", - "winner", - "criteria", - "evidence", - "explanation", - "confidence" - ], - "additionalProperties": false, - "properties": { - "case_id": { - "type": "string", - "minLength": 1 - }, - "model": { - "type": "string", - "minLength": 1 - }, - "provider": { - "type": "string", - "minLength": 1 - }, - "rubric_version": { - "type": "string", - "minLength": 1 - }, - "prompt_version": { - "type": "string", - "minLength": 1 - }, - "candidate_order": { - "type": "array", - "minItems": 2, - "maxItems": 2, - "items": { - "type": "string", - "minLength": 1 - } - }, - "randomization": { - "type": "object", - "required": ["method", "seed", "original_order", "candidate_order"], - "additionalProperties": false, - "properties": { - "method": { - "type": "string", - "minLength": 1 - }, - "seed": { - "anyOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "number" - } - ] - }, - "original_order": { - "type": "array", - "minItems": 2, - "maxItems": 2, - "items": { - "type": "string", - "minLength": 1 - } - }, - "candidate_order": { - "type": "array", - "minItems": 2, - "maxItems": 2, - "items": { - "type": "string", - "minLength": 1 - } - } - } - }, - "winner": { - "enum": ["candidate_a", "candidate_b", "tie", "unknown"] - }, - "criteria": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - }, - "evidence": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - }, - "explanation": { - "type": "string", - "minLength": 1 - }, - "confidence": { - "enum": ["low", "medium", "high"] - } - } -} diff --git a/packages/eval-kit/schemas/pointwise-judge-result.schema.json b/packages/eval-kit/schemas/pointwise-judge-result.schema.json deleted file mode 100644 index 5289e09..0000000 --- a/packages/eval-kit/schemas/pointwise-judge-result.schema.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/pointwise-judge-result.schema.json", - "title": "PointwiseJudgeResult", - "type": "object", - "required": [ - "case_id", - "model", - "provider", - "rubric_version", - "prompt_version", - "items" - ], - "additionalProperties": false, - "properties": { - "case_id": { - "type": "string", - "minLength": 1 - }, - "model": { - "type": "string", - "minLength": 1 - }, - "provider": { - "type": "string", - "minLength": 1 - }, - "rubric_version": { - "type": "string", - "minLength": 1 - }, - "prompt_version": { - "type": "string", - "minLength": 1 - }, - "items": { - "type": "array", - "minItems": 1, - "items": { - "type": "object", - "required": [ - "item_id", - "kind", - "verdict", - "severity", - "confidence", - "candidate_evidence", - "source_refs", - "explanation" - ], - "additionalProperties": false, - "properties": { - "item_id": { - "type": "string", - "pattern": "^(FACT|CTX)-[0-9]{3}$" - }, - "kind": { - "enum": ["fact", "boundary"] - }, - "verdict": { - "enum": ["covered", "partial", "missing", "contradicted", "unknown"] - }, - "severity": { - "enum": ["critical", "recommended"] - }, - "confidence": { - "enum": ["low", "medium", "high"] - }, - "candidate_evidence": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "source_refs": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "pattern": "^SRC-[0-9]{3}$" - } - }, - "explanation": { - "type": "string", - "minLength": 1 - } - }, - "allOf": [ - { - "if": { - "properties": { - "verdict": { - "enum": ["covered", "partial", "contradicted"] - } - }, - "required": ["verdict"] - }, - "then": { - "properties": { - "candidate_evidence": { - "type": "array", - "minItems": 1 - } - } - } - } - ] - } - } - } -} diff --git a/packages/eval-kit/schemas/result-manifest.v2.schema.json b/packages/eval-kit/schemas/result-manifest.v2.schema.json deleted file mode 100644 index 0954f35..0000000 --- a/packages/eval-kit/schemas/result-manifest.v2.schema.json +++ /dev/null @@ -1,154 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/result-manifest.v2.schema.json", - "title": "EvalKitResultManifestV2", - "type": "object", - "additionalProperties": false, - "required": [ - "schema_version", - "run_id", - "run_type", - "runner", - "case_ids", - "started_at", - "ended_at", - "duration_ms", - "status", - "git", - "command", - "tool_versions", - "artifacts", - "output_files" - ], - "properties": { - "schema_version": { - "const": "eval-kit.result-manifest.v2" - }, - "run_id": { - "type": "string", - "minLength": 1 - }, - "run_type": { - "type": "string", - "minLength": 1 - }, - "runner": { - "type": "object", - "additionalProperties": false, - "required": ["id", "version"], - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "version": { - "type": "string", - "minLength": 1 - } - } - }, - "case_ids": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "started_at": { - "type": "string", - "minLength": 1 - }, - "ended_at": { - "type": "string", - "minLength": 1 - }, - "duration_ms": { - "type": "integer", - "minimum": 0 - }, - "status": { - "enum": ["completed", "runner_failed"] - }, - "git": { - "type": "object", - "additionalProperties": false, - "required": ["commit"], - "properties": { - "commit": { - "type": "string", - "minLength": 1 - } - } - }, - "command": { - "type": "string", - "minLength": 1 - }, - "tool_versions": { - "type": "object", - "additionalProperties": { - "type": "string" - } - }, - "artifacts": { - "type": "array", - "items": { - "$ref": "https://agentic-workflow-kit.local/eval-kit/artifact.schema.json" - } - }, - "output_files": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "model": { - "type": "string" - }, - "provider": { - "type": "string" - }, - "model_provider": { - "type": "string" - }, - "reasoning_effort": { - "type": "string" - }, - "sandbox_mode": { - "type": "string" - }, - "approval_policy": { - "type": "string" - }, - "codex_auth_mode": { - "type": "string" - }, - "prompt_version": { - "type": "string" - }, - "rubric_version": { - "type": "string" - }, - "randomization": { - "type": "object" - }, - "provenance": { - "type": "object", - "additionalProperties": false, - "required": ["parent_run_ids"], - "properties": { - "parent_run_ids": { - "type": "array", - "items": { - "type": "string", - "minLength": 1 - } - }, - "trigger_source": { - "type": "string" - } - } - } - } -} diff --git a/packages/eval-kit/schemas/results-manifest.schema.json b/packages/eval-kit/schemas/results-manifest.schema.json deleted file mode 100644 index 3d830fc..0000000 --- a/packages/eval-kit/schemas/results-manifest.schema.json +++ /dev/null @@ -1,183 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/results-manifest.schema.json", - "title": "EvalResultManifest", - "type": "object", - "required": [ - "run_id", - "git_commit", - "command", - "case_ids", - "tool_versions", - "run_type", - "output_files" - ], - "additionalProperties": false, - "properties": { - "run_id": { - "type": "string", - "minLength": 1 - }, - "git_commit": { - "type": "string", - "minLength": 1 - }, - "command": { - "type": "string", - "minLength": 1 - }, - "case_ids": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - }, - "tool_versions": { - "type": "object", - "required": ["node"], - "additionalProperties": { - "type": "string" - }, - "properties": { - "node": { - "type": "string", - "minLength": 1 - } - } - }, - "run_type": { - "enum": [ - "deterministic", - "generation", - "judge", - "judge-coverage", - "outcome", - "manual-report" - ] - }, - "model_provider": { - "type": "string" - }, - "model": { - "type": "string" - }, - "provider": { - "type": "string" - }, - "reasoning_effort": { - "type": "string" - }, - "sandbox_mode": { - "type": "string" - }, - "approval_policy": { - "type": "string" - }, - "codex_auth_mode": { - "type": "string" - }, - "rubric_version": { - "type": "string" - }, - "prompt_version": { - "type": "string" - }, - "randomization": { - "type": "object", - "required": ["method", "seed", "original_order", "candidate_order"], - "additionalProperties": false, - "properties": { - "method": { - "type": "string", - "minLength": 1 - }, - "seed": { - "type": "string", - "minLength": 1 - }, - "original_order": { - "type": "array", - "minItems": 2, - "maxItems": 2, - "items": { - "type": "string", - "minLength": 1 - } - }, - "candidate_order": { - "type": "array", - "minItems": 2, - "maxItems": 2, - "items": { - "type": "string", - "minLength": 1 - } - } - } - }, - "output_files": { - "type": "array", - "minItems": 1, - "items": { - "type": "string", - "minLength": 1 - } - } - }, - "allOf": [ - { - "if": { - "properties": { - "run_type": { - "enum": ["generation", "judge"] - } - }, - "required": ["run_type"] - }, - "then": { - "properties": { - "model": true, - "provider": true, - "prompt_version": true, - "rubric_version": true, - "randomization": true - }, - "required": ["model", "provider", "prompt_version"] - } - }, - { - "if": { - "properties": { - "run_type": { - "const": "judge-coverage" - } - }, - "required": ["run_type"] - }, - "then": { - "properties": { - "model": true, - "provider": true, - "prompt_version": true, - "rubric_version": true - }, - "required": ["model", "provider", "prompt_version", "rubric_version"] - } - }, - { - "if": { - "properties": { - "run_type": { - "const": "judge" - } - }, - "required": ["run_type"] - }, - "then": { - "required": ["rubric_version", "randomization"] - } - } - ] -} diff --git a/packages/eval-kit/schemas/run-report.schema.json b/packages/eval-kit/schemas/run-report.schema.json deleted file mode 100644 index 6bdd091..0000000 --- a/packages/eval-kit/schemas/run-report.schema.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://agentic-workflow-kit.local/eval-kit/run-report.schema.json", - "title": "EvalKitRunReport", - "type": "object", - "additionalProperties": false, - "required": ["run_id", "status", "summary"], - "properties": { - "run_id": { - "type": "string", - "minLength": 1 - }, - "status": { - "type": "string", - "minLength": 1 - }, - "summary": { - "type": "string", - "minLength": 1 - } - } -} diff --git a/packages/eval-kit/src/args.mjs b/packages/eval-kit/src/args.mjs deleted file mode 100644 index f530d06..0000000 --- a/packages/eval-kit/src/args.mjs +++ /dev/null @@ -1,30 +0,0 @@ -export const parseArgs = (argv) => { - const args = {}; - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === "--") { - continue; - } - if (!arg.startsWith("--")) { - throw new Error(`unexpected positional argument: ${arg}`); - } - const name = arg.slice(2); - const next = argv[index + 1]; - if (!next || next.startsWith("--")) { - throw new Error(`missing value for --${name}`); - } - args[name] = next; - index += 1; - } - return args; -}; - -export const requireArg = (args, name) => { - if (!args[name]) { - throw new Error(`missing required argument --${name}`); - } - return args[name]; -}; - -export const defaultRunId = (prefix = "run") => - `${prefix}-${new Date().toISOString().replace(/[:.]/g, "-")}`; diff --git a/packages/eval-kit/src/artifacts.mjs b/packages/eval-kit/src/artifacts.mjs deleted file mode 100644 index e3e8de3..0000000 --- a/packages/eval-kit/src/artifacts.mjs +++ /dev/null @@ -1,114 +0,0 @@ -import { createHash } from "node:crypto"; -import fs from "node:fs"; -import path from "node:path"; - -const toPosixPath = (value) => value.split(path.sep).join("/"); - -export const sha256File = (filePath) => - createHash("sha256").update(fs.readFileSync(filePath)).digest("hex"); - -export const artifactRecord = ({ - role, - path: relativePath, - runDir, - mediaType, - redactionStatus = "public-safe", - encoding = "utf-8", - legacy = false, -}) => { - if (typeof role !== "string" || role.trim().length === 0) { - throw new Error("artifact role is required"); - } - if (path.isAbsolute(relativePath)) { - throw new Error(`artifact path must be relative: ${relativePath}`); - } - const absolutePath = path.resolve(runDir, relativePath); - const contained = - path.relative(runDir, absolutePath) === "" || - (!path.relative(runDir, absolutePath).startsWith("..") && - !path.isAbsolute(path.relative(runDir, absolutePath))); - if (!contained) { - throw new Error(`artifact path escapes run directory: ${relativePath}`); - } - const exists = fs.existsSync(absolutePath); - if (!legacy && !exists) { - throw new Error(`artifact does not exist: ${relativePath}`); - } - const stat = exists ? fs.statSync(absolutePath) : null; - return { - role, - path: toPosixPath(relativePath), - sha256: exists ? sha256File(absolutePath) : null, - size_bytes: stat?.size ?? 0, - media_type: mediaType, - encoding, - redaction_status: legacy ? "legacy-unknown" : redactionStatus, - }; -}; - -export const normalizeLegacyManifest = (manifest, runDir) => { - if (Array.isArray(manifest.artifacts)) { - return manifest; - } - const outputFiles = Array.isArray(manifest.output_files) - ? manifest.output_files - : []; - return { - schema_version: "eval-kit.result-manifest.v2", - run_id: manifest.run_id, - run_type: manifest.run_type, - runner: { - id: `legacy-${manifest.run_type ?? "unknown"}`, - version: "legacy", - }, - case_ids: manifest.case_ids ?? [], - started_at: manifest.started_at ?? new Date(0).toISOString(), - ended_at: manifest.ended_at ?? new Date(0).toISOString(), - duration_ms: manifest.duration_ms ?? 0, - status: "completed", - git: { - commit: manifest.git_commit ?? manifest.git?.commit ?? "unknown", - }, - command: manifest.command ?? "unknown", - tool_versions: manifest.tool_versions ?? {}, - artifacts: outputFiles.map((fileName) => - artifactRecord({ - role: "legacy_output", - path: fileName, - runDir, - mediaType: fileName.endsWith(".json") - ? "application/json" - : fileName.endsWith(".md") - ? "text/markdown" - : "application/octet-stream", - legacy: !fs.existsSync(path.resolve(runDir, fileName)), - }), - ), - output_files: outputFiles, - }; -}; - -export const writeManifest = ({ runDir, manifest, schemaRegistry }) => { - const outputFiles = [ - "manifest.json", - ...manifest.artifacts.map((artifact) => artifact.path), - ]; - const normalized = { - ...manifest, - output_files: manifest.output_files ?? outputFiles, - }; - schemaRegistry?.validateWithSchema?.( - "result-manifest.v2.schema.json", - normalized, - "result manifest", - ); - fs.mkdirSync(runDir, { recursive: true }); - fs.writeFileSync( - path.join(runDir, "manifest.json"), - `${JSON.stringify(normalized, null, 2)}\n`, - ); - return normalized; -}; - -export const writeResultBundle = ({ runDir, manifest, schemaRegistry }) => - writeManifest({ runDir, manifest, schemaRegistry }); diff --git a/packages/eval-kit/src/cli.mjs b/packages/eval-kit/src/cli.mjs deleted file mode 100644 index 9c00168..0000000 --- a/packages/eval-kit/src/cli.mjs +++ /dev/null @@ -1,211 +0,0 @@ -import path from "node:path"; -import { parseArgs, requireArg, defaultRunId } from "./args.mjs"; -import { loadConfig } from "./config.mjs"; -import { - runCase, - generateCandidate, - judgeCoverage, - judgePairwise, - compileReport, - validateFixtures, -} from "./sdk.mjs"; - -const requireEnabledMethod = (config, methodKey, commandName) => { - if (config.raw.methods?.[methodKey]?.enabled === false) { - throw new Error( - `${commandName} is disabled by methods.${methodKey}.enabled=false`, - ); - } -}; - -const printHelp = () => { - console.log(` -Usage: eval-kit [options] - -Commands: - run-case Grade a single case candidate deterministically - --case - --candidate - [--run-id ] - [--config ] - - generate Generate a candidate design using Promptfoo - --case - --model - --provider - --effort - --run-id - [--config ] - - judge-coverage Pointwise judge expected facts and boundaries coverage - --case - --candidate - --model - --provider - --effort - [--run-id ] - [--config ] - - judge-pairwise Pairwise compare two candidates - --case - --candidate-a - --candidate-b - --model - --provider - --effort - --seed - --run-id - [--config ] - - report Compile manual reports into a unified summary - --run-id - [--generate ] - [--deterministic ] - [--judge-coverage ] - [--judge ] - [--outcome ] - [--config ] - - validate-fixtures Validate case manifests and expected fixtures - [--config ] -`); -}; - -export const main = async () => { - const subcommand = process.argv[2]; - if (!subcommand || subcommand === "--help" || subcommand === "-h") { - printHelp(); - process.exit(1); - } - - const rawArgs = process.argv.slice(3); - let parsed; - try { - parsed = parseArgs(rawArgs); - } catch (error) { - console.error(`Error parsing arguments: ${error.message}`); - process.exit(1); - } - - const configPath = parsed.config ?? "eval-kit.config.json"; - let config; - try { - config = loadConfig(configPath); - } catch (error) { - console.error(`Error loading config: ${error.message}`); - process.exit(1); - } - - try { - switch (subcommand) { - case "run-case": { - const caseId = requireArg(parsed, "case"); - const candidate = requireArg(parsed, "candidate"); - const runId = parsed["run-id"] ?? defaultRunId("deterministic"); - - const result = await runCase({ - config, - caseId, - candidatePath: candidate, - runId, - }); - if (result.verdict === "red") { - console.error("Deterministic grading failed (verdict red)"); - process.exit(1); - } - break; - } - - case "generate": { - const caseId = requireArg(parsed, "case"); - const model = requireArg(parsed, "model"); - const provider = requireArg(parsed, "provider"); - const effort = requireArg(parsed, "effort"); - const runId = requireArg(parsed, "run-id"); - - await generateCandidate({ - config, - caseId, - model, - provider, - effort, - runId, - }); - break; - } - - case "judge-coverage": { - const caseId = requireArg(parsed, "case"); - const candidate = requireArg(parsed, "candidate"); - const model = requireArg(parsed, "model"); - const provider = requireArg(parsed, "provider"); - const effort = requireArg(parsed, "effort"); - const runId = parsed["run-id"] ?? defaultRunId("judge-coverage"); - - await judgeCoverage({ - config, - caseId, - candidatePath: candidate, - model, - provider, - effort, - runId, - }); - break; - } - - case "judge-pairwise": { - requireEnabledMethod(config, "judge_pairwise", "judge-pairwise"); - const caseId = requireArg(parsed, "case"); - const candidateA = requireArg(parsed, "candidate-a"); - const candidateB = requireArg(parsed, "candidate-b"); - const model = requireArg(parsed, "model"); - const provider = requireArg(parsed, "provider"); - const effort = requireArg(parsed, "effort"); - const seed = parseInt(requireArg(parsed, "seed"), 10); - const runId = requireArg(parsed, "run-id"); - - await judgePairwise({ - config, - caseId, - candidateAPath: candidateA, - candidateBPath: candidateB, - model, - provider, - effort, - seed, - runId, - }); - break; - } - - case "report": { - const runId = requireArg(parsed, "run-id"); - const runs = {}; - if (parsed.generate) runs.generate = parsed.generate; - if (parsed.deterministic) runs.deterministic = parsed.deterministic; - if (parsed["judge-coverage"]) - runs["judge-coverage"] = parsed["judge-coverage"]; - if (parsed.judge) runs.judge = parsed.judge; - if (parsed.outcome) runs.outcome = parsed.outcome; - - await compileReport({ config, runId, runs }); - break; - } - - case "validate-fixtures": { - await validateFixtures({ config }); - console.log("Fixtures validation completed successfully."); - break; - } - - default: - console.error(`Unknown command: ${subcommand}`); - printHelp(); - process.exit(1); - } - } catch (error) { - console.error(`Execution failed: ${error.message}`); - process.exit(1); - } -}; diff --git a/packages/eval-kit/src/config.mjs b/packages/eval-kit/src/config.mjs deleted file mode 100644 index 6b2f55b..0000000 --- a/packages/eval-kit/src/config.mjs +++ /dev/null @@ -1,149 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; -import { fileURLToPath, pathToFileURL } from "node:url"; - -import { createPathResolver } from "./paths.mjs"; -import { createSchemaRegistry } from "./schema.mjs"; - -const __dirname = path.dirname(fileURLToPath(import.meta.url)); -const defaultSchemasRoot = path.resolve(__dirname, "../schemas"); -const defaultPromptsRoot = path.resolve(__dirname, "../promptfoo"); - -const methodPromptKeyByTemplateKey = { - generation: "generate", - pointwise_judge: "judge_coverage", - pairwise_judge: "judge_pairwise", -}; - -const kitPromptAliases = new Set([ - "@eval-kit/generation", - "@eval-kit/pointwise", - "@eval-kit/pairwise", -]); - -export const loadConfig = (configFilePath) => { - const absoluteConfigPath = path.resolve(configFilePath); - if (!fs.existsSync(absoluteConfigPath)) { - throw new Error(`config file not found: ${configFilePath}`); - } - - const rawConfig = JSON.parse(fs.readFileSync(absoluteConfigPath, "utf8")); - - // Create a minimal registry to validate the config itself - const tempRegistry = createSchemaRegistry({ - schemaRoots: [defaultSchemasRoot], - }); - const config = tempRegistry.validateWithSchema( - "eval-kit.config.schema.json", - rawConfig, - "eval-kit config", - ); - - const configDir = path.dirname(absoluteConfigPath); - - // We need to determine the repo root. Let's find the closest .git or package.json up the tree. - let repoRoot = configDir; - while (repoRoot !== path.dirname(repoRoot)) { - if (fs.existsSync(path.join(repoRoot, ".git"))) { - break; - } - repoRoot = path.dirname(repoRoot); - } - - const pathResolver = createPathResolver({ - repoRoot, - suiteRoot: config.suite_root, - resultsRoot: config.results_root, - configDir, - }); - - // Resolve schema roots - const canonicalRoots = (config.schema_roots ?? []).map((root) => { - const abs = path.resolve(configDir, root); - return fs.existsSync(abs) ? fs.realpathSync(abs) : abs; - }); - const canonicalDefault = fs.existsSync(defaultSchemasRoot) - ? fs.realpathSync(defaultSchemasRoot) - : defaultSchemasRoot; - - const schemaRoots = Array.from( - new Set([canonicalDefault, ...canonicalRoots]), - ); - - const schemaRegistry = createSchemaRegistry({ schemaRoots }); - - // Expose helper to load dynamic adapters. - const loadModule = async (relativeOrAbsolutePath, label) => { - const resolvedPath = path.isAbsolute(relativeOrAbsolutePath) - ? relativeOrAbsolutePath - : path.resolve(pathResolver.suiteRoot, relativeOrAbsolutePath); - - if (!fs.existsSync(resolvedPath)) { - throw new Error(`${label} file does not exist: ${resolvedPath}`); - } - - try { - const fileUrl = pathToFileURL(resolvedPath).href; - return await import(fileUrl); - } catch (error) { - throw new Error( - `failed to load ${label} module from ${resolvedPath}: ${error.message}`, - ); - } - }; - - /** - * Resolve a prompt template path. If the consumer specifies a path in - * config.prompt_templates, resolve it relative to the suite root. Otherwise - * fall back to the kit-bundled prompt in packages/eval-kit/promptfoo/. - * - * @param {"generation"|"pointwise_judge"|"pairwise_judge"} key - template key - * @param {string} kitFallbackRelative - path relative to packages/eval-kit/promptfoo/ - * @returns {string} absolute path to the prompt template file - */ - const resolvePromptTemplate = (key, kitFallbackRelative) => { - const methodKey = methodPromptKeyByTemplateKey[key]; - const methodPrompt = methodKey ? config.methods?.[methodKey]?.prompt : null; - const consumerPath = config.prompt_templates?.[key] ?? methodPrompt; - if (consumerPath) { - if (consumerPath.startsWith("@eval-kit/")) { - if (!kitPromptAliases.has(consumerPath)) { - throw new Error(`unknown eval-kit prompt alias: ${consumerPath}`); - } - } else { - return pathResolver.resolveSuitePath( - consumerPath, - `prompt_templates.${key}`, - ); - } - } - const fallbackPath = path.resolve(defaultPromptsRoot, kitFallbackRelative); - if (!fs.existsSync(fallbackPath)) { - throw new Error( - `Kit-bundled prompt template not found: ${fallbackPath}. Specify prompt_templates.${key} in your eval-kit.config.json.`, - ); - } - return fallbackPath; - }; - - const resolveKitSchemaPath = (schemaFileName) => { - if (path.isAbsolute(schemaFileName) || schemaFileName.includes(path.sep)) { - throw new Error(`schema file name must not be a path: ${schemaFileName}`); - } - const schemaPath = path.resolve(defaultSchemasRoot, schemaFileName); - if (!fs.existsSync(schemaPath)) { - throw new Error(`Kit-bundled schema not found: ${schemaPath}`); - } - return schemaPath; - }; - - return { - raw: config, - configDir, - pathResolver, - schemaRegistry, - loadModule, - resolvePromptTemplate, - resolveKitSchemaPath, - }; -}; diff --git a/packages/eval-kit/src/grading.mjs b/packages/eval-kit/src/grading.mjs deleted file mode 100644 index dc4c849..0000000 --- a/packages/eval-kit/src/grading.mjs +++ /dev/null @@ -1,294 +0,0 @@ -export const normalize = (value) => - String(value ?? "") - .toLowerCase() - .replace(/[`*_~]/g, "") - .replace(/[‘’‛′']/g, "") - .replace(/[“”«»"]/g, "") - .replace(/[‐‑‒–—―]/g, "-") - .replace(/&/g, " and ") - .replace(/[^a-z0-9]+/g, " ") - .replace(/\s+/g, " ") - .trim(); - -const normalizedIncludes = (normalizedText, snippet) => { - const normalizedSnippet = normalize(snippet); - if (!normalizedSnippet) { - return false; - } - return ` ${normalizedText} `.includes(` ${normalizedSnippet} `); -}; - -const orderedTokenIncludes = (normalizedText, snippet) => { - const snippetTokens = normalize(snippet).split(" ").filter(Boolean); - if (snippetTokens.length === 0) { - return false; - } - - const textTokens = normalizedText.split(" ").filter(Boolean); - let nextSnippetIndex = 0; - for (const token of textTokens) { - if (token === snippetTokens[nextSnippetIndex]) { - nextSnippetIndex += 1; - } - if (nextSnippetIndex === snippetTokens.length) { - return true; - } - } - return false; -}; - -const findIncludedSnippet = (text, snippets = []) => { - const normalizedText = normalize(text); - return snippets.find((snippet) => - normalizedIncludes(normalizedText, snippet), - ); -}; - -const findMissingSnippets = (text, snippets = []) => { - const normalizedText = normalize(text); - return snippets.filter( - (snippet) => !normalizedIncludes(normalizedText, snippet), - ); -}; - -const listMarkerPattern = /^(\s*)(?:[-*]|\d+\.)\s+/; -const tableRowPattern = /^\s*\|/; -const ownershipPhrasePattern = /\b(?:owns?|reads?|does\s+not\s+own)\b/i; -const terminalSentencePattern = /[.!?)]$/; - -const isListItem = (line) => listMarkerPattern.test(line); - -const listIndent = (line) => { - const match = line.match(listMarkerPattern); - return match ? match[1].length : null; -}; - -const isNestedListItem = (line) => { - const indent = listIndent(line); - return indent !== null && indent > 0; -}; - -const isPlainListChild = (line) => - isListItem(line) && !ownershipPhrasePattern.test(line); - -const currentEndsWithColon = (currentLines) => - currentLines.at(-1)?.trim().endsWith(":") ?? false; - -const currentEndsSentence = (currentLines) => - terminalSentencePattern.test(currentLines.at(-1)?.trim() ?? ""); - -export const candidateSegments = (text) => { - const segments = []; - let currentLines = []; - let acceptsPlainListChildren = false; - - const flush = () => { - if (currentLines.length > 0) { - segments.push(currentLines.join("\n").trim()); - currentLines = []; - } - acceptsPlainListChildren = false; - }; - - const pushCurrentLine = (line) => { - currentLines.push(line); - if (line.trim().endsWith(":")) { - acceptsPlainListChildren = true; - } - }; - - for (const rawLine of String(text ?? "").split("\n")) { - const line = rawLine.trimEnd(); - if (line.trim().length === 0) { - flush(); - continue; - } - - if (tableRowPattern.test(line)) { - flush(); - segments.push(line.trim()); - continue; - } - - if (isNestedListItem(line)) { - pushCurrentLine(line); - continue; - } - - if (isListItem(line)) { - if ( - currentLines.length > 0 && - (currentEndsWithColon(currentLines) || acceptsPlainListChildren) && - isPlainListChild(line) - ) { - pushCurrentLine(line); - continue; - } - flush(); - pushCurrentLine(line); - continue; - } - - if (currentLines.length > 0 && !currentEndsSentence(currentLines)) { - pushCurrentLine(line); - continue; - } - - flush(); - pushCurrentLine(line); - } - - flush(); - return segments; -}; - -const findCoLocatedSnippetSegment = (text, snippets = []) => - candidateSegments(text).find((segment) => { - const normalizedSegment = normalize(segment); - return snippets.every( - (snippet) => - normalizedIncludes(normalizedSegment, snippet) || - orderedTokenIncludes(normalizedSegment, snippet), - ); - }); - -export const includesAny = (text, snippets = []) => - Boolean(findIncludedSnippet(text, snippets)); - -export const includesAll = (text, snippets = []) => - findMissingSnippets(text, snippets).length === 0; - -const findAcceptedAlternativeMatch = (candidateText, alternatives = []) => - alternatives.find((alternative) => { - const forbiddenHit = includesAny( - candidateText, - alternative.must_not_include_any ?? [], - ); - return ( - !forbiddenHit && - includesAll(candidateText, alternative.must_include_all ?? []) - ); - }); - -const findRequiredConceptMatch = (candidateText, conceptGroups = []) => { - if (conceptGroups.length === 0) { - return null; - } - - const matchedGroups = conceptGroups.filter((group) => - includesAny(candidateText, group.any_of ?? []), - ); - if (matchedGroups.length !== conceptGroups.length) { - return null; - } - - return matchedGroups; -}; - -const formatSnippets = (snippets) => - snippets.map((snippet) => `"${snippet}"`).join(", "); - -export const assessCoverage = (candidateText, expectation, options = {}) => { - const forbiddenSnippet = findIncludedSnippet( - candidateText, - expectation.must_not_include_any ?? [], - ); - if (forbiddenSnippet) { - return { - verdict: "contradicted", - evidence: `forbidden evidence hit: ${formatSnippets([forbiddenSnippet])}`, - }; - } - - const anyRequired = expectation.must_include_any ?? []; - const allRequired = - expectation.must_include_all ?? options.defaultMustIncludeAll ?? []; - const anyHit = - anyRequired.length === 0 - ? null - : findIncludedSnippet(candidateText, anyRequired); - const missingAll = findMissingSnippets(candidateText, allRequired); - const coLocatedSegment = - options.requireCoLocatedAll && allRequired.length > 0 - ? findCoLocatedSnippetSegment(candidateText, allRequired) - : null; - const requiredEvidenceHit = - options.requireCoLocatedAll && allRequired.length > 0 - ? Boolean(coLocatedSegment) - : missingAll.length === 0; - const exactRequiredHit = - (anyRequired.length === 0 || anyHit) && - requiredEvidenceHit && - (!options.requireCoLocatedAll || - allRequired.length === 0 || - Boolean(coLocatedSegment)); - - if (exactRequiredHit) { - const evidenceParts = []; - if (anyHit) { - evidenceParts.push(`any=${formatSnippets([anyHit])}`); - } - if (allRequired.length > 0) { - evidenceParts.push(`all=${formatSnippets(allRequired)}`); - } - if (coLocatedSegment) { - evidenceParts.push(`segment=${formatSnippets([coLocatedSegment])}`); - } - return { - verdict: "covered", - evidence: `exact evidence hit${evidenceParts.length > 0 ? `: ${evidenceParts.join("; ")}` : ""}`, - }; - } - - const alternativeMatch = findAcceptedAlternativeMatch( - candidateText, - expectation.accepted_alternatives, - ); - if (alternativeMatch) { - return { - verdict: "covered", - evidence: `accepted alternative matched: ${alternativeMatch.label}`, - }; - } - - const conceptMatch = findRequiredConceptMatch( - candidateText, - expectation.required_concepts, - ); - if (conceptMatch) { - return { - verdict: "covered", - evidence: `concept groups matched: ${conceptMatch.map((group) => group.label).join(", ")}`, - }; - } - - return { - verdict: "missing", - evidence: [ - "missing required evidence", - anyRequired.length > 0 && !anyHit - ? `missing_any=${formatSnippets(anyRequired)}` - : "", - missingAll.length > 0 ? `missing_all=${formatSnippets(missingAll)}` : "", - options.requireCoLocatedAll && - allRequired.length > 0 && - missingAll.length === 0 - ? `not_co_located=${formatSnippets(allRequired)}` - : "", - ] - .filter(Boolean) - .join("; "), - }; -}; - -export const gradeFacts = (candidateText, expectedFacts) => - expectedFacts.facts.map((fact) => { - const assessment = assessCoverage(candidateText, fact); - return { - id: fact.id, - kind: "fact", - severity: fact.severity, - verdict: assessment.verdict, - evidence: assessment.evidence, - }; - }); diff --git a/packages/eval-kit/src/index.mjs b/packages/eval-kit/src/index.mjs deleted file mode 100644 index 3378dbf..0000000 --- a/packages/eval-kit/src/index.mjs +++ /dev/null @@ -1,43 +0,0 @@ -export { defaultRunId, parseArgs, requireArg } from "./args.mjs"; -export { - assertContainedPath, - assertSafeId, - containsPath, - createPathResolver, - toPosixPath, -} from "./paths.mjs"; -export { createSchemaRegistry, validateWithSchema } from "./schema.mjs"; -export { - artifactRecord, - normalizeLegacyManifest, - sha256File, - writeManifest, - writeResultBundle, -} from "./artifacts.mjs"; -export { - extractPromptfooOutput, - parseJsonOutput, - runPromptfooRaw, -} from "./promptfoo.mjs"; -export { aggregateVerdict, criticalBlockerCount } from "./verdict.mjs"; - -export { loadConfig } from "./config.mjs"; -export { - normalize, - candidateSegments, - includesAny, - includesAll, - assessCoverage, - gradeFacts, -} from "./grading.mjs"; -export { - configuredCaseManifestPaths, - resolveCaseManifest, - discoverCaseIds, - runCase, - generateCandidate, - judgeCoverage, - judgePairwise, - compileReport, - validateFixtures, -} from "./sdk.mjs"; diff --git a/packages/eval-kit/src/paths.mjs b/packages/eval-kit/src/paths.mjs deleted file mode 100644 index 55b0470..0000000 --- a/packages/eval-kit/src/paths.mjs +++ /dev/null @@ -1,129 +0,0 @@ -import path from "node:path"; - -export const toPosixPath = (value) => value.split(path.sep).join("/"); - -export const containsPath = (basePath, childPath) => { - const relativePath = path.relative(basePath, childPath); - return ( - relativePath === "" || - (!relativePath.startsWith("..") && !path.isAbsolute(relativePath)) - ); -}; - -export const assertContainedPath = (basePath, childPath, label) => { - if (!containsPath(basePath, childPath)) { - throw new Error(`${label} escapes ${toPosixPath(basePath)}`); - } - return childPath; -}; - -export const assertSafeId = (value, label) => { - if (typeof value !== "string" || value.trim().length === 0) { - throw new Error(`${label} is required`); - } - if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(value)) { - throw new Error(`${label} must be an id, not a path: ${value}`); - } - return value; -}; - -const resolveConfiguredPath = ({ - configDir, - value, - label, - allowAbsolute = false, -}) => { - if (typeof value !== "string" || value.trim().length === 0) { - throw new Error(`${label} is required`); - } - if (path.isAbsolute(value) && !allowAbsolute) { - throw new Error(`${label} must be relative to the config file`); - } - return path.resolve(configDir, value); -}; - -export const createPathResolver = ({ - repoRoot, - suiteRoot, - resultsRoot, - configDir = repoRoot, -}) => { - const resolvedRepoRoot = path.resolve(repoRoot); - const resolvedSuiteRoot = resolveConfiguredPath({ - configDir, - value: suiteRoot, - label: "suite root", - allowAbsolute: true, - }); - const resolvedResultsRoot = resolveConfiguredPath({ - configDir, - value: resultsRoot, - label: "results root", - allowAbsolute: true, - }); - - assertContainedPath(resolvedRepoRoot, resolvedSuiteRoot, "suite root"); - assertContainedPath(resolvedRepoRoot, resolvedResultsRoot, "results root"); - - const relativeToRepo = (absolutePath) => - toPosixPath(path.relative(resolvedRepoRoot, absolutePath)); - const relativeToSuite = (absolutePath) => - toPosixPath(path.relative(resolvedSuiteRoot, absolutePath)); - const relativeToResults = (absolutePath) => - toPosixPath(path.relative(resolvedResultsRoot, absolutePath)); - - const resolveRunDir = (runId) => { - const safeRunId = assertSafeId(runId, "run id"); - return assertContainedPath( - resolvedResultsRoot, - path.resolve(resolvedResultsRoot, safeRunId), - `run id ${safeRunId}`, - ); - }; - - const resolveResultArtifact = (runDir, relativePath, label) => { - if (path.isAbsolute(relativePath)) { - throw new Error(`${label} must be relative to the run directory`); - } - return assertContainedPath( - runDir, - path.resolve(runDir, relativePath), - label, - ); - }; - - const resolveSuitePath = (relativePath, label = "suite path") => { - if (path.isAbsolute(relativePath)) { - throw new Error(`${label} must be relative to the suite root`); - } - return assertContainedPath( - resolvedSuiteRoot, - path.resolve(resolvedSuiteRoot, relativePath), - label, - ); - }; - - const resolveRepoPath = (relativePath, label = "repo path") => { - if (path.isAbsolute(relativePath)) { - throw new Error(`${label} must be relative to the repository root`); - } - return assertContainedPath( - resolvedRepoRoot, - path.resolve(resolvedRepoRoot, relativePath), - label, - ); - }; - - return { - repoRoot: resolvedRepoRoot, - suiteRoot: resolvedSuiteRoot, - resultsRoot: resolvedResultsRoot, - relativeToRepo, - relativeToSuite, - relativeToResults, - resolveRunDir, - resolveResultArtifact, - resolveSuitePath, - resolveRepoPath, - }; -}; diff --git a/packages/eval-kit/src/promptfoo.mjs b/packages/eval-kit/src/promptfoo.mjs deleted file mode 100644 index bf043ba..0000000 --- a/packages/eval-kit/src/promptfoo.mjs +++ /dev/null @@ -1,103 +0,0 @@ -import { execFileSync } from "node:child_process"; -import fs from "node:fs"; - -export const runPromptfooRaw = ({ - promptfooBin, - cwd, - configPath, - env = {}, -}) => { - if (!fs.existsSync(promptfooBin)) { - throw new Error(`promptfoo binary not found at ${promptfooBin}`); - } - return execFileSync(promptfooBin, ["eval", "-c", configPath], { - cwd, - encoding: "utf8", - stdio: ["ignore", "pipe", "pipe"], - env: { - ...process.env, - ...env, - PROMPTFOO_DISABLE_TELEMETRY: "1", - }, - }); -}; - -const knownOutputCandidates = (value) => [ - value?.results?.results?.[0]?.response?.output, - value?.results?.results?.[0]?.response?.text, - value?.results?.results?.[0]?.response?.content, - value?.results?.[0]?.response?.output, - value?.results?.[0]?.response?.text, - value?.results?.[0]?.response?.content, - value?.results?.[0]?.output, -]; - -const collectStrings = (value, strings = []) => { - if (typeof value === "string") { - strings.push(value); - return strings; - } - if (Array.isArray(value)) { - for (const item of value) { - collectStrings(item, strings); - } - return strings; - } - if (value && typeof value === "object") { - for (const item of Object.values(value)) { - collectStrings(item, strings); - } - } - return strings; -}; - -export const extractPromptfooOutput = (promptfooResults, selector) => { - if (!selector || selector === "known-output") { - const matches = knownOutputCandidates(promptfooResults).filter( - (candidate) => - (typeof candidate === "string" && candidate.trim()) || - (candidate && typeof candidate === "object"), - ); - if (matches.length !== 1) { - throw new Error( - `Promptfoo output selector known-output found ${matches.length} candidates`, - ); - } - const [candidate] = matches; - return typeof candidate === "string" - ? candidate.trim() - : JSON.stringify(candidate, null, 2); - } - if (selector === "legacy-longest-string") { - const strings = collectStrings(promptfooResults) - .map((item) => item.trim()) - .filter((item) => item.length > 0) - .sort((a, b) => b.length - a.length); - const markdownCandidate = - strings.find((item) => item.includes("#") || item.includes("```")) ?? - strings[0]; - if (!markdownCandidate) { - throw new Error("Promptfoo results did not include a model output"); - } - return markdownCandidate; - } - throw new Error(`unknown Promptfoo output selector: ${selector}`); -}; - -export const parseJsonOutput = (output) => { - const trimmed = String(output ?? "").trim(); - const withoutFence = trimmed - .replace(/^```(?:json)?\s*/i, "") - .replace(/\s*```$/i, "") - .trim(); - try { - return JSON.parse(withoutFence); - } catch { - const firstBrace = withoutFence.indexOf("{"); - const lastBrace = withoutFence.lastIndexOf("}"); - if (firstBrace >= 0 && lastBrace > firstBrace) { - return JSON.parse(withoutFence.slice(firstBrace, lastBrace + 1)); - } - throw new Error("model output was not valid JSON"); - } -}; diff --git a/packages/eval-kit/src/schema.mjs b/packages/eval-kit/src/schema.mjs deleted file mode 100644 index 8db22ea..0000000 --- a/packages/eval-kit/src/schema.mjs +++ /dev/null @@ -1,90 +0,0 @@ -import Ajv2020 from "ajv/dist/2020.js"; -import fs from "node:fs"; -import path from "node:path"; - -const readJson = (filePath) => JSON.parse(fs.readFileSync(filePath, "utf8")); - -const schemaFilesIn = (schemaRoot) => - fs - .readdirSync(schemaRoot) - .filter((fileName) => fileName.endsWith(".schema.json")) - .sort((a, b) => a.localeCompare(b)) - .map((fileName) => path.join(schemaRoot, fileName)); - -export const createSchemaRegistry = ({ schemaRoots }) => { - const ajv = new Ajv2020({ - allErrors: true, - strict: true, - loadSchema: undefined, - }); - const schemasByFileName = new Map(); - const seenIds = new Map(); - - for (const schemaRoot of schemaRoots.map((root) => path.resolve(root))) { - for (const schemaFile of schemaFilesIn(schemaRoot)) { - const schema = readJson(schemaFile); - if (typeof schema.$id === "string") { - if (seenIds.has(schema.$id)) { - throw new Error( - `duplicate schema $id ${schema.$id} in ${schemaFile} and ${seenIds.get(schema.$id)}`, - ); - } - seenIds.set(schema.$id, schemaFile); - ajv.addSchema(schema); - } - schemasByFileName.set(path.basename(schemaFile), { - schema, - schemaFile, - }); - } - } - - const compileByFileName = (schemaFileName) => { - const record = schemasByFileName.get(schemaFileName); - if (!record) { - throw new Error(`schema not found: ${schemaFileName}`); - } - if (record.schema.$id) { - const validate = ajv.getSchema(record.schema.$id); - if (!validate) { - throw new Error(`schema did not compile: ${schemaFileName}`); - } - return validate; - } - return ajv.compile(record.schema); - }; - - const validateWithSchema = (schemaIdOrFile, data, label) => { - const validate = schemaIdOrFile.endsWith(".schema.json") - ? compileByFileName(schemaIdOrFile) - : ajv.getSchema(schemaIdOrFile); - if (!validate) { - throw new Error(`${label} schema did not compile: ${schemaIdOrFile}`); - } - if (validate(data)) { - return data; - } - const details = (validate.errors ?? []) - .map((error) => `${error.instancePath || ""} ${error.message}`) - .join("; "); - throw new Error(`${label} failed schema validation: ${details}`); - }; - - return { - ajv, - validateWithSchema, - schemaIds: [...seenIds.keys()], - }; -}; - -export const validateWithSchema = ({ - schemaRoots, - schemaIdOrFile, - data, - label, -}) => - createSchemaRegistry({ schemaRoots }).validateWithSchema( - schemaIdOrFile, - data, - label, - ); diff --git a/packages/eval-kit/src/sdk.mjs b/packages/eval-kit/src/sdk.mjs deleted file mode 100644 index 083ad5e..0000000 --- a/packages/eval-kit/src/sdk.mjs +++ /dev/null @@ -1,1328 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; -import { createHash } from "node:crypto"; -import { execFileSync, spawnSync } from "node:child_process"; - -import { artifactRecord, writeManifest } from "./artifacts.mjs"; -import { - extractPromptfooOutput, - parseJsonOutput, - runPromptfooRaw, -} from "./promptfoo.mjs"; -import { aggregateVerdict, criticalBlockerCount } from "./verdict.mjs"; -import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs"; - -const DEFAULT_SANDBOX_MODE = "read-only"; -const DEFAULT_APPROVAL_POLICY = "never"; -const RANDOMIZATION_METHOD = "sha256-seed-parity-v1"; - -const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); - -const wildcardPattern = (pattern) => - new RegExp(`^${pattern.split("*").map(escapeRegExp).join(".*")}$`); - -const matchesAnyPattern = (value, patterns) => - patterns.some((pattern) => wildcardPattern(pattern).test(value)); - -export const configuredCaseManifestPaths = (config) => { - if (Array.isArray(config.raw.case_manifests)) { - return [...config.raw.case_manifests].sort(); - } - - const casesConfig = config.raw.cases; - if (!casesConfig) { - return []; - } - - const resolver = config.pathResolver; - const casesRoot = resolver.resolveSuitePath(casesConfig.root, "cases.root"); - if (!fs.existsSync(casesRoot)) { - throw new Error(`cases root does not exist: ${casesConfig.root}`); - } - if (!fs.statSync(casesRoot).isDirectory()) { - throw new Error(`cases root is not a directory: ${casesConfig.root}`); - } - - const include = casesConfig.include ?? ["*"]; - const exclude = casesConfig.exclude ?? []; - const entries = fs - .readdirSync(casesRoot, { withFileTypes: true }) - .filter((entry) => entry.isDirectory()) - .map((entry) => entry.name) - .filter((name) => matchesAnyPattern(name, include)) - .filter((name) => !matchesAnyPattern(name, exclude)) - .sort(); - - return entries.map((entryName) => - toPosixPath( - path.relative( - resolver.suiteRoot, - path.join(casesRoot, entryName, "case-manifest.json"), - ), - ), - ); -}; - -const loadAdapterModule = async (config, label) => { - const modulePath = config.raw.adapter ?? config.raw.hooks?.module; - if (!modulePath) { - throw new Error(`no adapter configured for ${label}`); - } - return config.loadModule(modulePath, label); -}; - -// Resolve case manifest in a generic way -export const resolveCaseManifest = (config, caseId) => { - const safeCaseId = assertSafeId(caseId, "case id"); - const resolver = config.pathResolver; - - const manifestPath = configuredCaseManifestPaths(config).find( - (manifestRelPath) => { - try { - const fullPath = resolver.resolveSuitePath( - manifestRelPath, - "case manifest", - ); - if (!fs.existsSync(fullPath)) return false; - const manifest = JSON.parse(fs.readFileSync(fullPath, "utf8")); - return manifest.case_id === safeCaseId; - } catch { - return false; - } - }, - ); - - if (!manifestPath) { - throw new Error(`case manifest not found for case: ${safeCaseId}`); - } - - const absoluteManifestPath = resolver.resolveSuitePath( - manifestPath, - "case manifest", - ); - const caseDir = path.dirname(absoluteManifestPath); - const manifest = JSON.parse(fs.readFileSync(absoluteManifestPath, "utf8")); - - config.schemaRegistry.validateWithSchema( - "case-manifest.schema.json", - manifest, - "case manifest", - ); - - const artifacts = manifest.artifacts.map((art) => { - if (path.isAbsolute(art.path)) { - throw new Error(`case artifact ${art.path} must be relative`); - } - const artAbsPath = assertContainedPath( - caseDir, - path.resolve(caseDir, art.path), - `case artifact ${art.path}`, - ); - if (!fs.existsSync(artAbsPath)) { - throw new Error( - `case artifact does not exist: ${art.path} at ${artAbsPath}`, - ); - } - return { - ...art, - absolutePath: artAbsPath, - }; - }); - - return { - caseId: manifest.case_id, - caseDir, - manifestPath: absoluteManifestPath, - manifest, - artifacts, - }; -}; - -export const discoverCaseIds = (config) => { - const resolver = config.pathResolver; - const caseIds = []; - for (const manifestRelPath of configuredCaseManifestPaths(config)) { - try { - const fullPath = resolver.resolveSuitePath( - manifestRelPath, - "case manifest", - ); - if (fs.existsSync(fullPath)) { - const manifest = JSON.parse(fs.readFileSync(fullPath, "utf8")); - if (manifest.case_id) { - caseIds.push(manifest.case_id); - } - } - } catch { - // Ignore invalid files during discovery - } - } - return caseIds.sort(); -}; - -const getToolVersions = (config) => { - const resolver = config.pathResolver; - const versions = { node: process.version }; - - const execVersion = (cmd, args, cwd = resolver.repoRoot) => { - try { - return execFileSync(cmd, args, { - cwd, - encoding: "utf8", - stdio: ["ignore", "pipe", "pipe"], - }).trim(); - } catch { - return "unavailable"; - } - }; - - versions.pnpm = execVersion("pnpm", ["--version"]); - versions.codex = execVersion("codex", ["--version"]); - - try { - const promptfooPath = path.resolve( - resolver.repoRoot, - "node_modules/.bin/promptfoo", - ); - if (fs.existsSync(promptfooPath)) { - versions.promptfoo = execVersion( - promptfooPath, - ["--version"], - path.dirname(promptfooPath), - ); - } else { - versions.promptfoo = "unavailable"; - } - } catch { - versions.promptfoo = "unavailable"; - } - - return versions; -}; - -const getGitCommit = (config) => { - try { - return execFileSync("git", ["rev-parse", "HEAD"], { - cwd: config.pathResolver.repoRoot, - encoding: "utf8", - }).trim(); - } catch { - return "unknown"; - } -}; - -const codexAuthMode = (config) => { - try { - const result = spawnSync("codex", ["login", "status"], { - cwd: config.pathResolver.repoRoot, - encoding: "utf8", - stdio: ["ignore", "pipe", "pipe"], - }); - if (result.status !== 0) { - throw new Error( - `Codex auth not ready: ${result.stderr || "run codex login"}`, - ); - } - const status = `${result.stdout ?? ""}\n${result.stderr ?? ""}`; - return status.toLowerCase().includes("chatgpt") ? "chatgpt-local" : "local"; - } catch (error) { - throw new Error(`Codex auth check failed: ${error.message}`); - } -}; - -const normalizeCodexProvider = (provider) => { - if (!provider || provider === "openai") { - return "openai:codex-app-server"; - } - if ( - provider === "openai:codex-app-server" || - provider.startsWith("openai:codex-app-server:") - ) { - return provider; - } - throw new Error( - `unsupported provider ${provider}; use openai or openai:codex-app-server`, - ); -}; - -const codexProviderId = ({ provider, model }) => { - const normalized = normalizeCodexProvider(provider); - if (normalized.startsWith("openai:codex-app-server:")) { - return normalized; - } - return `${normalized}:${model}`; -}; - -// Programmatic SDK implementation - -export const runCase = async ({ config, caseId, candidatePath, runId }) => { - const startedAt = new Date(); - const resolver = config.pathResolver; - const resolvedCandidatePath = path.resolve(resolver.repoRoot, candidatePath); - if (!fs.existsSync(resolvedCandidatePath)) { - throw new Error(`candidate path does not exist: ${candidatePath}`); - } - - const { artifacts, caseDir } = resolveCaseManifest(config, caseId); - const resultDir = resolver.resolveRunDir(runId); - const caseResultDir = path.join(resultDir, "cases", caseId); - - // Load configured grader and reporter - const graderName = - config.raw.methods?.deterministic?.grader ?? - config.raw.runner_defaults?.grader ?? - Object.keys(config.raw.graders ?? {})[0]; - const reporterName = - config.raw.methods?.deterministic?.reporter ?? - config.raw.runner_defaults?.reporter ?? - Object.keys(config.raw.reporters ?? {})[0]; - - const adapterModule = config.raw.adapter - ? await loadAdapterModule(config, "adapter") - : null; - const graderModule = - adapterModule ?? - (graderName && config.raw.graders?.[graderName] - ? await config.loadModule( - config.raw.graders[graderName], - `grader ${graderName}`, - ) - : null); - const reporterModule = - adapterModule ?? - (reporterName && config.raw.reporters?.[reporterName] - ? await config.loadModule( - config.raw.reporters[reporterName], - `reporter ${reporterName}`, - ) - : null); - - if (!graderModule) { - throw new Error("no grader configured in config"); - } - if (!reporterModule) { - throw new Error("no reporter configured in config"); - } - - // Resolve inputs for grader - const graderInputArtifacts = artifacts.filter( - (a) => a.role === "grader_input", - ); - const graderInputs = {}; - for (const art of graderInputArtifacts) { - const key = path - .basename(art.path, ".json") - .replace(/-([a-z])/g, (_, c) => c.toUpperCase()); - graderInputs[key] = JSON.parse(fs.readFileSync(art.absolutePath, "utf8")); - } - - const candidateText = fs.readFileSync(resolvedCandidatePath, "utf8"); - - // Call dynamic grader - // Standard interface: gradeCandidate({ candidateText, expectedFacts, expectedBoundaries, ... }) - const graderFunc = - graderModule.gradeTechnicalDesignCandidate || - graderModule.gradeCandidate || - graderModule.default; - if (typeof graderFunc !== "function") { - throw new Error(`grader module does not export a grading function`); - } - - const { findings, verdict } = graderFunc({ - candidateText, - ...graderInputs, - }); - - // Structural check of output grades - const grades = { - case_id: caseId, - verdict, - findings, - }; - config.schemaRegistry.validateWithSchema( - "grades.schema.json", - grades, - "grades", - ); - - fs.mkdirSync(caseResultDir, { recursive: true }); - fs.copyFileSync( - resolvedCandidatePath, - path.join(caseResultDir, "candidate.md"), - ); - fs.writeFileSync( - path.join(caseResultDir, "grader-output.json"), - JSON.stringify({ findings }, null, 2) + "\n", - ); - fs.writeFileSync( - path.join(resultDir, "grades.json"), - JSON.stringify(grades, null, 2) + "\n", - ); - - const outputFiles = [ - "manifest.json", - "grades.json", - "report.md", - `cases/${caseId}/candidate.md`, - `cases/${caseId}/grader-output.json`, - ]; - - // Call dynamic reporter - const reporterFunc = - reporterModule.renderDeterministicReport || - reporterModule.renderReport || - reporterModule.default; - if (typeof reporterFunc !== "function") { - throw new Error( - "reporter module does not export a report rendering function", - ); - } - - const reportContent = reporterFunc({ - caseId, - grades, - findings, - caseDir, - candidatePath: resolvedCandidatePath, - resolver, - }); - - fs.writeFileSync(path.join(resultDir, "report.md"), reportContent + "\n"); - - const endedAt = new Date(); - const artRecord = (role, fileName, mediaType) => - artifactRecord({ - role, - path: fileName, - runDir: resultDir, - mediaType, - redactionStatus: role.startsWith("raw_") ? "raw-local" : "public-safe", - }); - - writeManifest({ - runDir: resultDir, - schemaRegistry: config.schemaRegistry, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: runId, - run_type: "deterministic", - runner: { - id: `${config.raw.suite_id}-eval-case`, - version: "0.0.0", - }, - case_ids: [caseId], - started_at: startedAt.toISOString(), - ended_at: endedAt.toISOString(), - duration_ms: endedAt.getTime() - startedAt.getTime(), - status: "completed", - git: { commit: getGitCommit(config) }, - command: process.argv.join(" "), - tool_versions: getToolVersions(config), - artifacts: [ - artRecord("grades", "grades.json", "application/json"), - artRecord("report", "report.md", "text/markdown"), - artRecord( - "candidate_markdown", - `cases/${caseId}/candidate.md`, - "text/markdown", - ), - artRecord( - "grader_output", - `cases/${caseId}/grader-output.json`, - "application/json", - ), - ], - output_files: outputFiles, - }, - }); - - return { verdict, findings, resultDir }; -}; - -export const generateCandidate = async ({ - config, - caseId, - model, - provider: providerArg, - effort, - runId, -}) => { - const startedAt = new Date(); - const resolver = config.pathResolver; - const authMode = codexAuthMode(config); - const provider = normalizeCodexProvider(providerArg); - - const { artifacts, caseDir } = resolveCaseManifest(config, caseId); - const resultDir = resolver.resolveRunDir(runId); - const caseResultDir = path.join(resultDir, "cases", caseId); - fs.mkdirSync(caseResultDir, { recursive: true }); - - const promptTemplatePath = config.resolvePromptTemplate( - "generation", - "generation.prompt.md", - ); - const promptText = fs.readFileSync(promptTemplatePath, "utf8"); - - // Load custom adapter - const hooks = await loadAdapterModule(config, "adapter"); - - if (typeof hooks.resolveGenerationVars !== "function") { - throw new Error( - "adapter module does not export resolveGenerationVars function", - ); - } - - const vars = await hooks.resolveGenerationVars({ - caseId, - caseDir, - artifacts, - resolver, - }); - - // Create promptfoo config - const promptfooConfigPath = path.join(resultDir, "promptfooconfig.json"); - const promptfooResultsPath = path.join(resultDir, "promptfoo-results.json"); - const promptfooReportPath = path.join(resultDir, "promptfoo-report.html"); - - const promptfooConfig = { - description: `${config.raw.suite_id} generation eval for ${caseId}`, - prompts: [promptText], - providers: [ - { - id: codexProviderId({ provider, model }), - config: { - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - model_reasoning_effort: effort, - }, - }, - ], - tests: [ - { - vars: { - case_id: caseId, - ...vars, - }, - }, - ], - outputPath: [promptfooResultsPath, promptfooReportPath], - }; - - fs.writeFileSync( - promptfooConfigPath, - JSON.stringify(promptfooConfig, null, 2) + "\n", - ); - - const promptfooBin = path.resolve( - resolver.repoRoot, - "node_modules/.bin/promptfoo", - ); - runPromptfooRaw({ - promptfooBin, - cwd: resolver.repoRoot, - configPath: promptfooConfigPath, - env: { PROMPTFOO_DISABLE_TELEMETRY: "1" }, - }); - - const promptfooResults = JSON.parse( - fs.readFileSync(promptfooResultsPath, "utf8"), - ); - const candidateText = extractPromptfooOutput(promptfooResults).trim(); - if (!candidateText) { - throw new Error("Promptfoo generation output was empty"); - } - - const candidatePath = path.join(caseResultDir, "candidate.md"); - fs.writeFileSync(candidatePath, `${candidateText}\n`); - - fs.writeFileSync( - path.join(resultDir, "report.md"), - [ - `# Generation Eval Report: ${caseId}`, - "", - `Model: ${model}`, - `Provider: ${provider}`, - `Auth mode: ${authMode}`, - "", - "## Outputs", - "", - `- Candidate: ${resolver.relativeToRepo(candidatePath)}`, - `- Promptfoo JSON: ${resolver.relativeToRepo(promptfooResultsPath)}`, - `- Promptfoo HTML: ${resolver.relativeToRepo(promptfooReportPath)}`, - ].join("\n") + "\n", - ); - - const endedAt = new Date(); - const artRecord = (role, fileName, mediaType) => - artifactRecord({ - role, - path: fileName, - runDir: resultDir, - mediaType, - redactionStatus: role.startsWith("raw_") ? "raw-local" : "public-safe", - }); - - writeManifest({ - runDir: resultDir, - schemaRegistry: config.schemaRegistry, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: runId, - run_type: "generation", - runner: { - id: `${config.raw.suite_id}-generate`, - version: "0.0.0", - }, - case_ids: [caseId], - started_at: startedAt.toISOString(), - ended_at: endedAt.toISOString(), - duration_ms: endedAt.getTime() - startedAt.getTime(), - status: "completed", - git: { commit: getGitCommit(config) }, - command: process.argv.join(" "), - tool_versions: getToolVersions(config), - model_provider: codexProviderId({ provider, model }), - model, - provider, - reasoning_effort: effort, - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - codex_auth_mode: authMode, - artifacts: [ - artRecord("report", "report.md", "text/markdown"), - artRecord( - "promptfoo_config", - "promptfooconfig.json", - "application/json", - ), - artRecord( - "raw_promptfoo_results", - "promptfoo-results.json", - "application/json", - ), - artRecord( - "promptfoo_html_report", - "promptfoo-report.html", - "text/html", - ), - artRecord( - "candidate_markdown", - `cases/${caseId}/candidate.md`, - "text/markdown", - ), - ], - output_files: [ - "manifest.json", - "report.md", - "promptfooconfig.json", - "promptfoo-results.json", - "promptfoo-report.html", - `cases/${caseId}/candidate.md`, - ], - }, - }); - - return { candidatePath, resultDir }; -}; - -export const judgeCoverage = async ({ - config, - caseId, - candidatePath, - model, - provider: providerArg, - effort, - runId, -}) => { - const startedAt = new Date(); - const resolver = config.pathResolver; - const authMode = codexAuthMode(config); - const provider = normalizeCodexProvider(providerArg); - - const resolvedCandidatePath = path.resolve(resolver.repoRoot, candidatePath); - const { artifacts, caseDir } = resolveCaseManifest(config, caseId); - const resultDir = resolver.resolveRunDir(runId); - const caseResultDir = path.join(resultDir, "cases", caseId); - fs.mkdirSync(caseResultDir, { recursive: true }); - - const promptTemplatePath = config.resolvePromptTemplate( - "pointwise_judge", - "judges/pointwise.prompt.md", - ); - const promptText = fs.readFileSync(promptTemplatePath, "utf8"); - - // Find version from prompt file (looks for "Prompt version: `...`" or similar) - const extractVersion = (text, label) => { - const matches = [ - ...text.matchAll(new RegExp(`${label}:\\s*\\\`([^\\\`]+)\\\``, "g")), - ]; - const match = matches.find((cand) => !cand[1].includes("{{")); - if (!match) throw new Error(`Could not find ${label} in prompt`); - return match[1]; - }; - - const promptVersion = extractVersion(promptText, "Prompt version"); - const rubricVersion = extractVersion(promptText, "Rubric version"); - - const outputSchemaPath = config.resolveKitSchemaPath( - "pointwise-judge-result.schema.json", - ); - const outputSchema = JSON.parse(fs.readFileSync(outputSchemaPath, "utf8")); - - // Load custom adapter - const hooks = await loadAdapterModule(config, "adapter"); - - if (typeof hooks.resolvePointwiseVars !== "function") { - throw new Error( - "adapter module does not export resolvePointwiseVars function", - ); - } - - const vars = await hooks.resolvePointwiseVars({ - caseId, - caseDir, - artifacts, - candidateContent: fs.readFileSync(resolvedCandidatePath, "utf8"), - candidatePath: resolvedCandidatePath, - promptVersion, - rubricVersion, - model, - provider, - resolver, - }); - - const promptfooConfigPath = path.join(resultDir, "promptfooconfig.json"); - const promptfooResultsPath = path.join(resultDir, "promptfoo-results.json"); - const promptfooReportPath = path.join(resultDir, "promptfoo-report.html"); - - // Strip unsupported allOf fields from schema for API compatibility - const cleanSchema = (val) => { - if (Array.isArray(val)) return val.map(cleanSchema); - if (val && typeof val === "object") { - return Object.fromEntries( - Object.entries(val) - .filter(([k]) => k !== "allOf") - .map(([k, v]) => [k, cleanSchema(v)]), - ); - } - return val; - }; - - const promptfooConfig = { - description: `${config.raw.suite_id} pointwise judge for ${caseId}`, - prompts: [promptTemplatePath], - providers: [ - { - id: codexProviderId({ provider, model }), - config: { - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - model_reasoning_effort: effort, - output_schema: cleanSchema(outputSchema), - }, - }, - ], - tests: [{ vars }], - outputPath: [promptfooResultsPath, promptfooReportPath], - }; - - fs.writeFileSync( - promptfooConfigPath, - JSON.stringify(promptfooConfig, null, 2) + "\n", - ); - - const promptfooBin = path.resolve( - resolver.repoRoot, - "node_modules/.bin/promptfoo", - ); - runPromptfooRaw({ - promptfooBin, - cwd: resolver.repoRoot, - configPath: promptfooConfigPath, - env: { PROMPTFOO_DISABLE_TELEMETRY: "1" }, - }); - - const promptfooResults = JSON.parse( - fs.readFileSync(promptfooResultsPath, "utf8"), - ); - const rawResult = parseJsonOutput(extractPromptfooOutput(promptfooResults)); - - const result = config.schemaRegistry.validateWithSchema( - "pointwise-judge-result.schema.json", - rawResult, - "pointwise judge result", - ); - - // Validate properties match run config - if (result.case_id !== caseId) throw new Error("case_id mismatch in result"); - if (result.model !== model) throw new Error("model mismatch in result"); - - // Post-process pointwise items if the adapter provides custom canonicalization - let finalResult = result; - if (typeof hooks.canonicalizeExpectedItemMetadata === "function") { - finalResult = { - ...result, - items: hooks.canonicalizeExpectedItemMetadata( - result.items, - vars._expectedItemsForCanonicalization, - ), - }; - } - - const pointwiseResultPath = path.join(resultDir, "pointwise-result.json"); - fs.writeFileSync( - pointwiseResultPath, - JSON.stringify(finalResult, null, 2) + "\n", - ); - - const counts = { - covered: 0, - partial: 0, - missing: 0, - contradicted: 0, - unknown: 0, - }; - for (const item of finalResult.items) { - counts[item.verdict] = (counts[item.verdict] ?? 0) + 1; - } - - fs.writeFileSync( - path.join(resultDir, "report.md"), - [ - `# Pointwise Judge Report: ${caseId}`, - "", - `Model: ${model}`, - `Provider: ${provider}`, - `Auth mode: ${authMode}`, - `Prompt version: ${promptVersion}`, - `Rubric version: ${rubricVersion}`, - `Candidate: ${resolver.relativeToRepo(resolvedCandidatePath)}`, - "", - "## Coverage Summary", - "", - `- covered: ${counts.covered}`, - `- partial: ${counts.partial}`, - `- missing: ${counts.missing}`, - `- contradicted: ${counts.contradicted}`, - `- unknown: ${counts.unknown}`, - "", - "## Item Results", - "", - ...finalResult.items.map( - (item) => - `- ${item.item_id} (${item.kind}, ${item.severity}): ${item.verdict} [${item.confidence}]${item.candidate_evidence?.length > 0 ? ` evidence=${item.candidate_evidence.join(" | ")}` : ""} explanation=${item.explanation}`, - ), - ].join("\n") + "\n", - ); - - const endedAt = new Date(); - const artRecord = (role, fileName, mediaType) => - artifactRecord({ - role, - path: fileName, - runDir: resultDir, - mediaType, - redactionStatus: role.startsWith("raw_") ? "raw-local" : "public-safe", - }); - - writeManifest({ - runDir: resultDir, - schemaRegistry: config.schemaRegistry, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: runId, - run_type: "judge-coverage", - runner: { - id: `${config.raw.suite_id}-pointwise-judge`, - version: "0.0.0", - }, - case_ids: [caseId], - started_at: startedAt.toISOString(), - ended_at: endedAt.toISOString(), - duration_ms: endedAt.getTime() - startedAt.getTime(), - status: "completed", - git: { commit: getGitCommit(config) }, - command: process.argv.join(" "), - tool_versions: getToolVersions(config), - model_provider: codexProviderId({ provider, model }), - model, - provider, - reasoning_effort: effort, - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - codex_auth_mode: authMode, - prompt_version: promptVersion, - rubric_version: rubricVersion, - artifacts: [ - artRecord("report", "report.md", "text/markdown"), - artRecord( - "pointwise_result", - "pointwise-result.json", - "application/json", - ), - artRecord( - "promptfoo_config", - "promptfooconfig.json", - "application/json", - ), - artRecord( - "raw_promptfoo_results", - "promptfoo-results.json", - "application/json", - ), - artRecord( - "promptfoo_html_report", - "promptfoo-report.html", - "text/html", - ), - ], - output_files: [ - "manifest.json", - "report.md", - "pointwise-result.json", - "promptfooconfig.json", - "promptfoo-results.json", - "promptfoo-report.html", - ], - }, - }); - - return { resultDir, finalResult }; -}; - -export const judgePairwise = async ({ - config, - caseId, - candidateAPath, - candidateBPath, - model, - provider: providerArg, - effort, - seed, - runId, -}) => { - const startedAt = new Date(); - const resolver = config.pathResolver; - const authMode = codexAuthMode(config); - const provider = normalizeCodexProvider(providerArg); - - const resolvedCandidateAPath = path.resolve( - resolver.repoRoot, - candidateAPath, - ); - const resolvedCandidateBPath = path.resolve( - resolver.repoRoot, - candidateBPath, - ); - - const { artifacts, caseDir } = resolveCaseManifest(config, caseId); - const resultDir = resolver.resolveRunDir(runId); - fs.mkdirSync(resultDir, { recursive: true }); - - const promptTemplatePath = config.resolvePromptTemplate( - "pairwise_judge", - "judges/pairwise.prompt.md", - ); - const promptText = fs.readFileSync(promptTemplatePath, "utf8"); - - const extractVersion = (text, label) => { - const matches = [ - ...text.matchAll(new RegExp(`${label}:\\s*\\\`([^\\\`]+)\\\``, "g")), - ]; - const match = matches.find((cand) => !cand[1].includes("{{")); - if (!match) throw new Error(`Could not find ${label} in prompt`); - return match[1]; - }; - - const promptVersion = extractVersion(promptText, "Prompt version"); - const rubricVersion = extractVersion(promptText, "Rubric version"); - - const outputSchemaPath = config.resolveKitSchemaPath( - "pairwise-result.schema.json", - ); - const outputSchema = JSON.parse(fs.readFileSync(outputSchemaPath, "utf8")); - - // Randomize candidate order using sha256 of paths and seed - const originalOrder = ["candidate_a", "candidate_b"]; - const digest = createHash("sha256") - .update( - JSON.stringify({ - seed, - case_id: caseId, - candidate_a: resolver.relativeToRepo(resolvedCandidateAPath), - candidate_b: resolver.relativeToRepo(resolvedCandidateBPath), - }), - ) - .digest("hex"); - const shouldSwap = Number.parseInt(digest.slice(0, 2), 16) % 2 === 1; - const candidateOrder = shouldSwap - ? ["candidate_b", "candidate_a"] - : ["candidate_a", "candidate_b"]; - - const randomizedOrder = { - method: RANDOMIZATION_METHOD, - seed, - original_order: originalOrder, - candidate_order: candidateOrder, - }; - - // Load custom adapter - const hooks = await loadAdapterModule(config, "adapter"); - - if (typeof hooks.resolvePairwiseVars !== "function") { - throw new Error( - "adapter module does not export resolvePairwiseVars function", - ); - } - - const vars = await hooks.resolvePairwiseVars({ - caseId, - caseDir, - artifacts, - candidateAContent: fs.readFileSync(resolvedCandidateAPath, "utf8"), - candidateBContent: fs.readFileSync(resolvedCandidateBPath, "utf8"), - candidateAPath: resolvedCandidateAPath, - candidateBPath: resolvedCandidateBPath, - promptVersion, - rubricVersion, - model, - provider, - randomizedOrder, - resolver, - }); - - const promptfooConfigPath = path.join(resultDir, "promptfooconfig.json"); - const promptfooResultsPath = path.join(resultDir, "promptfoo-results.json"); - const promptfooReportPath = path.join(resultDir, "promptfoo-report.html"); - - const cleanSchema = (val) => { - if (Array.isArray(val)) return val.map(cleanSchema); - if (val && typeof val === "object") { - return Object.fromEntries( - Object.entries(val) - .filter(([k]) => k !== "allOf") - .map(([k, v]) => [k, cleanSchema(v)]), - ); - } - return val; - }; - - const promptfooConfig = { - description: `${config.raw.suite_id} pairwise judge for ${caseId}`, - prompts: [promptTemplatePath], - providers: [ - { - id: codexProviderId({ provider, model }), - config: { - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - model_reasoning_effort: effort, - output_schema: cleanSchema(outputSchema), - }, - }, - ], - tests: [{ vars }], - outputPath: [promptfooResultsPath, promptfooReportPath], - }; - - fs.writeFileSync( - promptfooConfigPath, - JSON.stringify(promptfooConfig, null, 2) + "\n", - ); - - const promptfooBin = path.resolve( - resolver.repoRoot, - "node_modules/.bin/promptfoo", - ); - runPromptfooRaw({ - promptfooBin, - cwd: resolver.repoRoot, - configPath: promptfooConfigPath, - env: { PROMPTFOO_DISABLE_TELEMETRY: "1" }, - }); - - const promptfooResults = JSON.parse( - fs.readFileSync(promptfooResultsPath, "utf8"), - ); - const rawResult = parseJsonOutput(extractPromptfooOutput(promptfooResults)); - - const pairwiseOutput = config.schemaRegistry.validateWithSchema( - "pairwise-result.schema.json", - rawResult, - "pairwise judge output", - ); - - const assertMatchingValue = (label, actual, expected) => { - if (actual !== expected) { - throw new Error( - `${label} mismatch in pairwise judge output: expected ${expected}, received ${actual}`, - ); - } - }; - - const assertMatchingArray = (label, actual, expected) => { - if (JSON.stringify(actual) !== JSON.stringify(expected)) { - throw new Error( - `${label} mismatch in pairwise judge output: expected ${JSON.stringify(expected)}, received ${JSON.stringify(actual)}`, - ); - } - }; - - assertMatchingValue("case_id", pairwiseOutput.case_id, caseId); - assertMatchingValue("model", pairwiseOutput.model, model); - assertMatchingValue("provider", pairwiseOutput.provider, provider); - assertMatchingValue( - "prompt_version", - pairwiseOutput.prompt_version, - promptVersion, - ); - assertMatchingValue( - "rubric_version", - pairwiseOutput.rubric_version, - rubricVersion, - ); - assertMatchingArray( - "candidate_order", - pairwiseOutput.candidate_order, - randomizedOrder.candidate_order, - ); - assertMatchingValue( - "randomization.method", - pairwiseOutput.randomization.method, - randomizedOrder.method, - ); - assertMatchingValue( - "randomization.seed", - String(pairwiseOutput.randomization.seed), - String(randomizedOrder.seed), - ); - assertMatchingArray( - "randomization.original_order", - pairwiseOutput.randomization.original_order, - randomizedOrder.original_order, - ); - assertMatchingArray( - "randomization.candidate_order", - pairwiseOutput.randomization.candidate_order, - randomizedOrder.candidate_order, - ); - - let winner = pairwiseOutput.winner; - if (shouldSwap && (winner === "candidate_a" || winner === "candidate_b")) { - winner = winner === "candidate_a" ? "candidate_b" : "candidate_a"; - } - - const pairwiseResult = { - ...pairwiseOutput, - candidate_order: originalOrder, - randomization: randomizedOrder, - winner, - }; - - config.schemaRegistry.validateWithSchema( - "pairwise-result.schema.json", - pairwiseResult, - "pairwise result", - ); - - const pairwiseResultPath = path.join(resultDir, "pairwise-result.json"); - fs.writeFileSync( - pairwiseResultPath, - JSON.stringify(pairwiseResult, null, 2) + "\n", - ); - - fs.writeFileSync( - path.join(resultDir, "report.md"), - [ - `# Pairwise Judge Report: ${caseId}`, - "", - `Model: ${model}`, - `Provider: ${provider}`, - `Auth mode: ${authMode}`, - `Prompt version: ${promptVersion}`, - `Rubric version: ${rubricVersion}`, - `Seed: ${seed}`, - `Randomization swapped: ${shouldSwap}`, - "", - `Winner: ${winner}`, - `Confidence: ${pairwiseResult.confidence}`, - `Explanation: ${pairwiseResult.explanation}`, - ].join("\n") + "\n", - ); - - const endedAt = new Date(); - const artRecord = (role, fileName, mediaType) => - artifactRecord({ - role, - path: fileName, - runDir: resultDir, - mediaType, - redactionStatus: role.startsWith("raw_") ? "raw-local" : "public-safe", - }); - - writeManifest({ - runDir: resultDir, - schemaRegistry: config.schemaRegistry, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: runId, - run_type: "judge", - runner: { - id: `${config.raw.suite_id}-pairwise-judge`, - version: "0.0.0", - }, - case_ids: [caseId], - started_at: startedAt.toISOString(), - ended_at: endedAt.toISOString(), - duration_ms: endedAt.getTime() - startedAt.getTime(), - status: "completed", - git: { commit: getGitCommit(config) }, - command: process.argv.join(" "), - tool_versions: getToolVersions(config), - model_provider: codexProviderId({ provider, model }), - model, - provider, - reasoning_effort: effort, - sandbox_mode: DEFAULT_SANDBOX_MODE, - approval_policy: DEFAULT_APPROVAL_POLICY, - codex_auth_mode: authMode, - prompt_version: promptVersion, - rubric_version: rubricVersion, - artifacts: [ - artRecord("report", "report.md", "text/markdown"), - artRecord( - "pairwise_result", - "pairwise-result.json", - "application/json", - ), - artRecord( - "promptfoo_config", - "promptfooconfig.json", - "application/json", - ), - artRecord( - "raw_promptfoo_results", - "promptfoo-results.json", - "application/json", - ), - artRecord( - "promptfoo_html_report", - "promptfoo-report.html", - "text/html", - ), - ], - output_files: [ - "manifest.json", - "report.md", - "pairwise-result.json", - "promptfooconfig.json", - "promptfoo-results.json", - "promptfoo-report.html", - ], - }, - }); - - return { resultDir, pairwiseResult }; -}; - -export const compileReport = async ({ config, runId, runs }) => { - const startedAt = new Date(); - const resolver = config.pathResolver; - const resultDir = resolver.resolveRunDir(runId); - fs.mkdirSync(resultDir, { recursive: true }); - - // Load custom adapter - const hooks = await loadAdapterModule(config, "adapter"); - - if (typeof hooks.compileReport !== "function") { - throw new Error("adapter module does not export compileReport function"); - } - - const { - reportContent, - caseIds, - artifacts: extraArtifacts, - outputFiles: extraOutputFiles, - } = await hooks.compileReport({ - config, - runId, - runs, - resultDir, - resolver, - }); - - fs.writeFileSync(path.join(resultDir, "report.md"), reportContent + "\n"); - - const endedAt = new Date(); - const artRecord = (role, fileName, mediaType) => - artifactRecord({ - role, - path: fileName, - runDir: resultDir, - mediaType, - redactionStatus: role.startsWith("raw_") ? "raw-local" : "public-safe", - }); - - const artifacts = [ - artRecord("report", "report.md", "text/markdown"), - ...extraArtifacts.map((art) => - artRecord(art.role, art.path, art.mediaType), - ), - ]; - - writeManifest({ - runDir: resultDir, - schemaRegistry: config.schemaRegistry, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: runId, - run_type: "manual-report", - runner: { - id: `${config.raw.suite_id}-manual-report`, - version: "0.0.0", - }, - case_ids: caseIds, - started_at: startedAt.toISOString(), - ended_at: endedAt.toISOString(), - duration_ms: endedAt.getTime() - startedAt.getTime(), - status: "completed", - git: { commit: getGitCommit(config) }, - command: process.argv.join(" "), - tool_versions: getToolVersions(config), - artifacts, - output_files: ["manifest.json", "report.md", ...extraOutputFiles], - }, - }); - - return { resultDir }; -}; - -export const validateFixtures = async ({ config }) => { - const manifests = []; - for (const manifestRelPath of configuredCaseManifestPaths(config)) { - const fullPath = config.pathResolver.resolveSuitePath( - manifestRelPath, - "case manifest", - ); - if (!fs.existsSync(fullPath)) { - throw new Error(`configured case manifest not found: ${manifestRelPath}`); - } - const manifest = JSON.parse(fs.readFileSync(fullPath, "utf8")); - config.schemaRegistry.validateWithSchema( - "case-manifest.schema.json", - manifest, - `case manifest ${manifestRelPath}`, - ); - manifests.push({ manifest, fullPath, relativePath: manifestRelPath }); - } - - // Load custom adapter - const hooks = - (config.raw.adapter ?? config.raw.hooks?.module) - ? await loadAdapterModule(config, "adapter") - : {}; - - if (typeof hooks.validateFixtures === "function") { - await hooks.validateFixtures({ config, manifests }); - } -}; diff --git a/packages/eval-kit/src/verdict.mjs b/packages/eval-kit/src/verdict.mjs deleted file mode 100644 index 6a7841e..0000000 --- a/packages/eval-kit/src/verdict.mjs +++ /dev/null @@ -1,32 +0,0 @@ -export const aggregateVerdict = (findings, policy) => { - const blockingSeverities = new Set(policy.blocking_severities ?? []); - const blockingVerdicts = new Set(policy.blocking_verdicts ?? []); - const nonGreenVerdicts = new Set(policy.non_green_verdicts ?? []); - const yellowVerdict = policy.yellow_verdict ?? "yellow"; - const redVerdict = policy.red_verdict ?? "red"; - const greenVerdict = policy.green_verdict ?? "green"; - - if ( - findings.some( - (finding) => - blockingSeverities.has(finding.severity) && - blockingVerdicts.has(finding.verdict), - ) - ) { - return redVerdict; - } - if (findings.some((finding) => nonGreenVerdicts.has(finding.verdict))) { - return yellowVerdict; - } - return greenVerdict; -}; - -export const criticalBlockerCount = (findings, policy) => { - const blockingSeverities = new Set(policy.blocking_severities ?? []); - const blockingVerdicts = new Set(policy.blocking_verdicts ?? []); - return findings.filter( - (finding) => - blockingSeverities.has(finding.severity) && - blockingVerdicts.has(finding.verdict), - ).length; -}; diff --git a/packages/eval-kit/tests/artifacts.test.mjs b/packages/eval-kit/tests/artifacts.test.mjs deleted file mode 100644 index 5c28fba..0000000 --- a/packages/eval-kit/tests/artifacts.test.mjs +++ /dev/null @@ -1,91 +0,0 @@ -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - -import { describe, expect, it } from "vitest"; - -import { - artifactRecord, - normalizeLegacyManifest, - sha256File, - writeManifest, -} from "../src/index.mjs"; - -describe("eval-kit artifacts", () => { - it("records artifact metadata and rejects escapes", () => { - const runDir = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-run-")); - fs.writeFileSync(path.join(runDir, "report.md"), "# Report\n"); - - const artifact = artifactRecord({ - role: "report", - path: "report.md", - runDir, - mediaType: "text/markdown", - }); - - expect(artifact.sha256).toBe(sha256File(path.join(runDir, "report.md"))); - expect(artifact.size_bytes).toBeGreaterThan(0); - expect(() => - artifactRecord({ - role: "report", - path: "../report.md", - runDir, - mediaType: "text/markdown", - }), - ).toThrow(/escapes/); - }); - - it("normalizes legacy output_files manifests", () => { - const runDir = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-legacy-")); - fs.writeFileSync(path.join(runDir, "manifest.json"), "{}\n"); - const normalized = normalizeLegacyManifest( - { - run_id: "legacy-run", - run_type: "deterministic", - git_commit: "abc123", - command: "pnpm eval", - case_ids: ["case-a"], - tool_versions: { node: "v26.4.0" }, - output_files: ["manifest.json", "missing.md"], - }, - runDir, - ); - - expect(normalized.schema_version).toBe("eval-kit.result-manifest.v2"); - expect(normalized.artifacts).toHaveLength(2); - expect(normalized.artifacts[0].sha256).toMatch(/^[a-f0-9]{64}$/); - expect(normalized.artifacts[1].sha256).toBeNull(); - }); - - it("defaults output_files to manifest plus artifact paths", () => { - const runDir = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-write-")); - fs.writeFileSync(path.join(runDir, "report.md"), "# Report\n"); - const written = writeManifest({ - runDir, - manifest: { - schema_version: "eval-kit.result-manifest.v2", - run_id: "run-001", - run_type: "deterministic", - runner: { id: "test", version: "0.0.0" }, - case_ids: ["case-a"], - started_at: "2026-07-01T00:00:00.000Z", - ended_at: "2026-07-01T00:00:01.000Z", - duration_ms: 1000, - status: "completed", - git: { commit: "abc123" }, - command: "pnpm test", - tool_versions: { node: "v26.4.0" }, - artifacts: [ - artifactRecord({ - role: "report", - path: "report.md", - runDir, - mediaType: "text/markdown", - }), - ], - }, - }); - - expect(written.output_files).toEqual(["manifest.json", "report.md"]); - }); -}); diff --git a/packages/eval-kit/tests/case-manifest.test.mjs b/packages/eval-kit/tests/case-manifest.test.mjs deleted file mode 100644 index 623bbd0..0000000 --- a/packages/eval-kit/tests/case-manifest.test.mjs +++ /dev/null @@ -1,168 +0,0 @@ -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; -import { describe, expect, it } from "vitest"; - -import { - loadConfig, - discoverCaseIds, - resolveCaseManifest, -} from "@agentic-workflow-kit/eval-kit"; - -const __filename = fileURLToPath(import.meta.url); -const packageRoot = path.resolve(path.dirname(__filename), ".."); -const repoRoot = path.resolve(packageRoot, "../.."); -const configPath = path.join(repoRoot, "evals", "eval-kit.config.json"); - -const expectedCaseIds = [ - "case-aerial-delivery-shipping-v1", - "case-cloudevents-core-contract-v1", - "case-customer-credit-order-saga-v1", - "case-fineract-loan-lifecycle-v1", - "case-kubernetes-sidecar-containers-v1", - "case-openfeature-evaluation-api-v1", - "case-tiny-laundry-pickup-v1", -]; - -const expectedArtifacts = [ - { role: "generation_visible", path: "product.md" }, - { role: "generation_visible", path: "source-map.md" }, - { role: "grader_input", path: "expected-facts.json" }, - { role: "grader_input", path: "expected-boundaries.json" }, - { role: "semantic_reference", path: "reference-design.md" }, - { role: "semantic_reference", path: "rubric.md" }, - { role: "maintainer_only", path: "grader-notes.md" }, - { role: "provenance", path: "provenance.md" }, -]; - -const writeJson = (filePath, value) => { - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, `${JSON.stringify(value, null, 2)}\n`); -}; - -const writeText = (filePath, value) => { - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, value); -}; - -const createTempSuite = ({ caseId, artifactPath }) => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-case-")); - const configFile = path.join(root, "evals", "eval-kit.config.json"); - const caseDir = path.join(root, "evals", "cases", caseId); - - writeJson(configFile, { - schema_version: "eval-kit.config.v1", - suite_id: "test-suite", - suite_root: ".", - results_root: "results", - adapter: "adapter.mjs", - cases: { - root: "cases", - include: ["case-*"], - }, - methods: {}, - }); - writeJson(path.join(caseDir, "case-manifest.json"), { - schema_version: "test.case-manifest.v1", - case_id: caseId, - artifacts: [{ role: "input", path: artifactPath }], - }); - return { configFile, caseDir }; -}; - -describe("technical-design case manifest resolver", () => { - it("uses the simplified consumer config surface", () => { - const config = loadConfig(configPath); - - expect(config.raw.adapter).toBe("adapter.mjs"); - expect(config.raw.schema_roots).toBeUndefined(); - expect(config.raw.case_manifests).toBeUndefined(); - expect(config.raw.cases).toEqual({ - root: "cases", - include: ["case-*-v1"], - }); - expect(config.raw.methods.deterministic).toEqual({ - enabled: true, - grader: "facts-boundaries", - reporter: "markdown", - }); - expect(config.raw.methods.judge_coverage).toEqual({ - enabled: true, - prompt: "@eval-kit/pointwise", - rubric: "case:rubric.md", - }); - expect(config.raw.methods.judge_pairwise).toEqual({ - enabled: false, - }); - }); - - it("discovers all expected case ids from manifests", () => { - const config = loadConfig(configPath); - const manifestCaseIds = discoverCaseIds(config); - expect(manifestCaseIds).toEqual(expectedCaseIds); - }); - - it("resolves manifest-backed cases to the fixed case directory artifacts", () => { - const config = loadConfig(configPath); - for (const caseId of expectedCaseIds) { - const resolved = resolveCaseManifest(config, caseId); - - expect(resolved.caseId).toBe(caseId); - expect(resolved.manifest.schema_version).toBe( - "technical-design.case-manifest.v1", - ); - expect(resolved.manifest.artifacts).toEqual(expectedArtifacts); - expect(resolved.manifest.grading).toEqual({ - grader: "technical-design-facts-boundaries", - co_located_boundaries: true, - allow_reference_design_as_ground_truth: false, - }); - expect(resolved.manifest.grounding).toEqual({ - source_ref_allowlist: ["product.md", "source-map.md"], - }); - expect(resolved.caseDir).toBe( - path.join(repoRoot, "evals", "cases", caseId), - ); - expect(resolved.artifacts.map((artifact) => artifact.path)).toEqual( - expectedArtifacts.map((artifact) => artifact.path), - ); - } - }); - - it("rejects manifest artifact paths that escape the case directory", () => { - const { configFile } = createTempSuite({ - caseId: "case-escape", - artifactPath: "../outside.json", - }); - writeText( - path.join(path.dirname(configFile), "cases", "outside.json"), - "{}\n", - ); - const config = loadConfig(configFile); - - expect(() => resolveCaseManifest(config, "case-escape")).toThrow( - /case artifact \.\.\/outside\.json escapes/, - ); - }); - - it("rejects absolute manifest artifact paths", () => { - const caseId = "case-absolute"; - const temp = createTempSuite({ - caseId, - artifactPath: "placeholder.md", - }); - const absoluteArtifactPath = path.join(temp.caseDir, "product.md"); - writeText(absoluteArtifactPath, "product\n"); - writeJson(path.join(temp.caseDir, "case-manifest.json"), { - schema_version: "test.case-manifest.v1", - case_id: caseId, - artifacts: [{ role: "input", path: absoluteArtifactPath }], - }); - const config = loadConfig(temp.configFile); - - expect(() => resolveCaseManifest(config, caseId)).toThrow( - /case artifact .* must be relative/, - ); - }); -}); diff --git a/packages/eval-kit/tests/cli.test.mjs b/packages/eval-kit/tests/cli.test.mjs deleted file mode 100644 index ccd89dc..0000000 --- a/packages/eval-kit/tests/cli.test.mjs +++ /dev/null @@ -1,28 +0,0 @@ -import { spawnSync } from "node:child_process"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { describe, expect, it } from "vitest"; - -const __filename = fileURLToPath(import.meta.url); -const packageRoot = path.resolve(path.dirname(__filename), ".."); -const repoRoot = path.resolve(packageRoot, "../.."); -const configPath = path.join(repoRoot, "evals", "eval-kit.config.json"); -const cliPath = path.resolve(packageRoot, "bin/eval-kit.mjs"); - -describe("eval-kit CLI", () => { - it("fails closed when pairwise judging is disabled by config", () => { - const result = spawnSync( - process.execPath, - [cliPath, "judge-pairwise", "--config", configPath], - { - cwd: packageRoot, - encoding: "utf8", - stdio: ["ignore", "pipe", "pipe"], - }, - ); - - expect(result.status).toBe(1); - expect(result.stderr).toContain("judge-pairwise is disabled"); - }); -}); diff --git a/packages/eval-kit/tests/paths.test.mjs b/packages/eval-kit/tests/paths.test.mjs deleted file mode 100644 index f7ada59..0000000 --- a/packages/eval-kit/tests/paths.test.mjs +++ /dev/null @@ -1,36 +0,0 @@ -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - -import { describe, expect, it } from "vitest"; - -import { assertSafeId, createPathResolver } from "../src/index.mjs"; - -describe("eval-kit path helpers", () => { - it("rejects path-shaped ids", () => { - expect(assertSafeId("case-a", "case id")).toBe("case-a"); - expect(() => assertSafeId("../case-a", "case id")).toThrow("must be an id"); - expect(() => assertSafeId("nested/case", "case id")).toThrow( - "must be an id", - ); - }); - - it("contains suite, result, run, and artifact paths", () => { - const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-paths-")); - const resolver = createPathResolver({ - repoRoot, - configDir: repoRoot, - suiteRoot: "suites/example", - resultsRoot: "suites/example/results", - }); - const runDir = resolver.resolveRunDir("run-001"); - - expect(resolver.relativeToRepo(runDir)).toBe( - "suites/example/results/run-001", - ); - expect(() => resolver.resolveRunDir("../run-001")).toThrow("must be an id"); - expect(() => - resolver.resolveResultArtifact(runDir, "../manifest.json", "artifact"), - ).toThrow(/escapes/); - }); -}); diff --git a/packages/eval-kit/tests/promptfoo.test.mjs b/packages/eval-kit/tests/promptfoo.test.mjs deleted file mode 100644 index dbdb99e..0000000 --- a/packages/eval-kit/tests/promptfoo.test.mjs +++ /dev/null @@ -1,78 +0,0 @@ -import { describe, expect, it } from "vitest"; - -import { extractPromptfooOutput, parseJsonOutput } from "../src/index.mjs"; - -describe("eval-kit promptfoo helpers", () => { - it("extracts a single response output", () => { - expect( - extractPromptfooOutput({ - results: [{ response: { output: "candidate" } }], - }), - ).toBe("candidate"); - }); - - it("prefers response output over longer prompt or source strings", () => { - expect( - extractPromptfooOutput({ - results: [ - { - vars: { - prompt: - "# Long prompt\n\nThis source text is much longer than the model output.", - }, - response: { output: "candidate" }, - }, - ], - }), - ).toBe("candidate"); - }); - - it("extracts alternate response fields and nested promptfoo rows", () => { - expect( - extractPromptfooOutput({ - results: [{ response: { text: "candidate text" } }], - }), - ).toBe("candidate text"); - expect( - extractPromptfooOutput({ - results: [{ response: { content: "candidate content" } }], - }), - ).toBe("candidate content"); - expect( - extractPromptfooOutput({ - results: { results: [{ response: { output: "nested" } }] }, - }), - ).toBe("nested"); - }); - - it("serializes object outputs and fails closed on ambiguity", () => { - expect( - extractPromptfooOutput({ - results: [{ response: { output: { ok: true } } }], - }), - ).toBe(JSON.stringify({ ok: true }, null, 2)); - expect(() => - extractPromptfooOutput({ - results: [{ response: { output: "candidate", text: "other" } }], - }), - ).toThrow(/found 2 candidates/); - }); - - it("keeps legacy longest-string fallback explicit", () => { - expect( - extractPromptfooOutput( - { - results: [{ response: { output: "small" } }], - nested: "# Much longer markdown output", - }, - "legacy-longest-string", - ), - ).toBe("# Much longer markdown output"); - }); - - it("parses fenced json output", () => { - expect(parseJsonOutput('```json\n{"ok":true}\n```')).toEqual({ - ok: true, - }); - }); -}); diff --git a/packages/eval-kit/tests/report-cli.test.mjs b/packages/eval-kit/tests/report-cli.test.mjs deleted file mode 100644 index 97cdf90..0000000 --- a/packages/eval-kit/tests/report-cli.test.mjs +++ /dev/null @@ -1,142 +0,0 @@ -import { execFileSync } from "node:child_process"; -import fs from "node:fs"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; - -import { afterEach, describe, expect, it } from "vitest"; - -import { loadConfig } from "@agentic-workflow-kit/eval-kit"; - -const __filename = fileURLToPath(import.meta.url); -const packageRoot = path.resolve(path.dirname(__filename), ".."); -const repoRoot = path.resolve(packageRoot, "../.."); -const configPath = path.join(repoRoot, "evals", "eval-kit.config.json"); - -const config = loadConfig(configPath); -const resolver = config.pathResolver; - -const sourceRunId = "unit-v2-source-run"; -const reportRunId = "unit-v2-manual-report"; - -const removeRun = (runId) => { - fs.rmSync(resolver.resolveRunDir(runId), { recursive: true, force: true }); -}; - -const writeJson = (filePath, value) => { - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, `${JSON.stringify(value, null, 2)}\n`); -}; - -const writeText = (filePath, value) => { - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, value); -}; - -describe("manual report runner", () => { - afterEach(() => { - removeRun(sourceRunId); - removeRun(reportRunId); - }); - - it("preserves source commit provenance from v2 manifests and writes v2 output", () => { - removeRun(sourceRunId); - removeRun(reportRunId); - - const sourceRunDir = resolver.resolveRunDir(sourceRunId); - writeJson(path.join(sourceRunDir, "manifest.json"), { - schema_version: "eval-kit.result-manifest.v2", - run_id: sourceRunId, - run_type: "deterministic", - runner: { - id: "unit-deterministic", - version: "0.0.0", - }, - case_ids: ["case-tiny-laundry-pickup-v1"], - started_at: "2026-07-01T00:00:00.000Z", - ended_at: "2026-07-01T00:00:01.000Z", - duration_ms: 1000, - status: "completed", - git: { - commit: "feedface1234", - }, - command: "pnpm eval:case", - tool_versions: { - node: "v26.0.0", - }, - artifacts: [ - { - role: "report", - path: "report.md", - sha256: - "0000000000000000000000000000000000000000000000000000000000000000", - size_bytes: 14, - media_type: "text/markdown", - encoding: "utf-8", - redaction_status: "public-safe", - }, - ], - output_files: ["manifest.json", "grades.json", "report.md"], - }); - writeJson(path.join(sourceRunDir, "grades.json"), { - case_id: "case-tiny-laundry-pickup-v1", - verdict: "green", - findings: [], - }); - writeText(path.join(sourceRunDir, "report.md"), "Verdict: green\n"); - - execFileSync( - process.execPath, - [ - path.resolve(repoRoot, "packages/eval-kit/bin/eval-kit.mjs"), - "report", - "--run-id", - reportRunId, - "--deterministic", - sourceRunId, - "--config", - configPath, - ], - { - cwd: packageRoot, - stdio: "pipe", - }, - ); - - const reportRunDir = resolver.resolveRunDir(reportRunId); - const report = fs.readFileSync( - path.join(reportRunDir, "report.md"), - "utf8", - ); - const manifest = JSON.parse( - fs.readFileSync(path.join(reportRunDir, "manifest.json"), "utf8"), - ); - - expect(report).toContain("Deterministic Run results"); - expect(report).toContain("Verdict: green"); - expect(manifest.schema_version).toBe("eval-kit.result-manifest.v2"); - expect(manifest.run_type).toBe("manual-report"); - expect(manifest.git.commit).toMatch(/\S/); - expect(manifest.git.commit).not.toBe("unknown"); - expect(manifest.output_files).toEqual([ - "manifest.json", - "report.md", - "deterministic_grades.json", - ]); - expect(manifest.artifacts).toEqual([ - expect.objectContaining({ - role: "report", - path: "report.md", - sha256: expect.stringMatching(/^[a-f0-9]{64}$/), - media_type: "text/markdown", - redaction_status: "public-safe", - }), - expect.objectContaining({ - role: "deterministic_grades", - path: "deterministic_grades.json", - sha256: expect.stringMatching(/^[a-f0-9]{64}$/), - media_type: "application/json", - redaction_status: "public-safe", - }), - ]); - }); -}); diff --git a/packages/eval-kit/tests/schema.test.mjs b/packages/eval-kit/tests/schema.test.mjs deleted file mode 100644 index 4ac1a33..0000000 --- a/packages/eval-kit/tests/schema.test.mjs +++ /dev/null @@ -1,212 +0,0 @@ -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - -import { describe, expect, it } from "vitest"; - -import { createSchemaRegistry, loadConfig } from "../src/index.mjs"; - -const writeJson = (filePath, value) => { - fs.mkdirSync(path.dirname(filePath), { recursive: true }); - fs.writeFileSync(filePath, `${JSON.stringify(value, null, 2)}\n`); -}; - -describe("eval-kit schema registry", () => { - it("keeps bundled schema ids under the eval-kit namespace", () => { - const registry = createSchemaRegistry({ - schemaRoots: [path.resolve(import.meta.dirname, "../schemas")], - }); - - expect(registry.schemaIds).toEqual( - expect.arrayContaining([ - "https://agentic-workflow-kit.local/eval-kit/eval-kit.config.schema.json", - "https://agentic-workflow-kit.local/eval-kit/pairwise-result.schema.json", - ]), - ); - expect( - registry.schemaIds.filter((schemaId) => - schemaId.includes("/technical-design/evals/"), - ), - ).toEqual([]); - }); - - it("fails on duplicate schema ids", () => { - const rootA = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-schema-a-")); - const rootB = fs.mkdtempSync(path.join(os.tmpdir(), "eval-kit-schema-b-")); - const schema = { - $schema: "https://json-schema.org/draft/2020-12/schema", - $id: "https://example.test/duplicate.schema.json", - type: "object", - }; - writeJson(path.join(rootA, "a.schema.json"), schema); - writeJson(path.join(rootB, "b.schema.json"), schema); - - expect(() => createSchemaRegistry({ schemaRoots: [rootA, rootB] })).toThrow( - /duplicate schema/, - ); - }); - - it("validates result manifests with portable schemas", () => { - const registry = createSchemaRegistry({ - schemaRoots: [path.resolve(import.meta.dirname, "../schemas")], - }); - expect(() => - registry.validateWithSchema( - "result-manifest.v2.schema.json", - { - schema_version: "eval-kit.result-manifest.v2", - run_id: "run-001", - run_type: "deterministic", - runner: { id: "test", version: "0.0.0" }, - case_ids: ["case-a"], - started_at: "2026-07-01T00:00:00.000Z", - ended_at: "2026-07-01T00:00:01.000Z", - duration_ms: 1000, - status: "completed", - git: { commit: "abc123" }, - command: "pnpm test", - tool_versions: { node: "v26.4.0" }, - artifacts: [], - output_files: [], - }, - "manifest", - ), - ).not.toThrow(); - }); - - it("accepts current model-run metadata fields", () => { - const registry = createSchemaRegistry({ - schemaRoots: [path.resolve(import.meta.dirname, "../schemas")], - }); - expect(() => - registry.validateWithSchema( - "result-manifest.v2.schema.json", - { - schema_version: "eval-kit.result-manifest.v2", - run_id: "run-001", - run_type: "generation", - runner: { id: "generate", version: "0.0.0" }, - case_ids: ["case-a"], - started_at: "2026-07-01T00:00:00.000Z", - ended_at: "2026-07-01T00:00:01.000Z", - duration_ms: 1000, - status: "completed", - git: { commit: "abc123" }, - command: "pnpm eval", - tool_versions: { node: "v26.4.0" }, - artifacts: [], - output_files: ["manifest.json"], - model: "gpt-5.4", - provider: "openai:codex-app-server", - model_provider: "openai:codex-app-server:gpt-5.4", - reasoning_effort: "medium", - sandbox_mode: "read-only", - approval_policy: "never", - codex_auth_mode: "chatgpt-local", - prompt_version: "generation-prompt-v1", - }, - "manifest", - ), - ).not.toThrow(); - }); - - it("resolves bundled prompt and schema fallbacks for consumer configs", () => { - const config = loadConfig( - path.resolve(import.meta.dirname, "../../../evals/eval-kit.config.json"), - ); - - expect( - config.resolvePromptTemplate("generation", "generation.prompt.md"), - ).toBe( - path.resolve(import.meta.dirname, "../promptfoo/generation.prompt.md"), - ); - expect( - config.resolveKitSchemaPath("pointwise-judge-result.schema.json"), - ).toBe( - path.resolve( - import.meta.dirname, - "../schemas/pointwise-judge-result.schema.json", - ), - ); - }); - - it("resolves method prompt aliases from simplified configs", () => { - const config = loadConfig( - path.resolve(import.meta.dirname, "../../../evals/eval-kit.config.json"), - ); - - expect( - config.resolvePromptTemplate( - "pointwise_judge", - "judges/pointwise.prompt.md", - ), - ).toBe( - path.resolve( - import.meta.dirname, - "../promptfoo/judges/pointwise.prompt.md", - ), - ); - }); - - it("validates pointwise judge results", () => { - const registry = createSchemaRegistry({ - schemaRoots: [path.resolve(import.meta.dirname, "../schemas")], - }); - expect(() => - registry.validateWithSchema( - "pointwise-judge-result.schema.json", - { - case_id: "case-a", - model: "gpt-5.4", - provider: "openai:codex-app-server", - rubric_version: "v1", - prompt_version: "v1", - items: [ - { - item_id: "FACT-001", - kind: "fact", - verdict: "covered", - severity: "critical", - confidence: "high", - candidate_evidence: ["evidence"], - source_refs: ["SRC-001"], - explanation: "explain", - }, - ], - }, - "result", - ), - ).not.toThrow(); - }); - - it("validates pairwise judge results with randomized candidate order", () => { - const registry = createSchemaRegistry({ - schemaRoots: [path.resolve(import.meta.dirname, "../schemas")], - }); - expect(() => - registry.validateWithSchema( - "pairwise-result.schema.json", - { - case_id: "case-a", - model: "gpt-5.4", - provider: "openai:codex-app-server", - rubric_version: "judge-rubric-v1", - prompt_version: "pairwise-prompt-v1", - candidate_order: ["candidate_b", "candidate_a"], - randomization: { - method: "sha256-seed-parity-v1", - seed: 1234, - original_order: ["candidate_a", "candidate_b"], - candidate_order: ["candidate_b", "candidate_a"], - }, - winner: "candidate_a", - criteria: ["Source preservation"], - evidence: ["candidate_a cites SRC-001"], - explanation: "candidate_a preserves the required source fact.", - confidence: "high", - }, - "pairwise result", - ), - ).not.toThrow(); - }); -}); diff --git a/packages/eval-kit/tests/verdict.test.mjs b/packages/eval-kit/tests/verdict.test.mjs deleted file mode 100644 index 3b36ffc..0000000 --- a/packages/eval-kit/tests/verdict.test.mjs +++ /dev/null @@ -1,49 +0,0 @@ -import { describe, expect, it } from "vitest"; - -import { aggregateVerdict, criticalBlockerCount } from "../src/index.mjs"; - -const policy = { - blocking_severities: ["critical"], - blocking_verdicts: ["missing", "contradicted", "invented"], - non_green_verdicts: ["missing", "contradicted", "invented", "unknown"], - red_verdict: "red", - yellow_verdict: "yellow", - green_verdict: "green", -}; - -describe("eval-kit verdict aggregation", () => { - it("uses configurable blocker and non-green policy", () => { - expect( - aggregateVerdict([{ severity: "critical", verdict: "missing" }], policy), - ).toBe("red"); - expect( - aggregateVerdict( - [{ severity: "recommended", verdict: "unknown" }], - policy, - ), - ).toBe("yellow"); - expect( - aggregateVerdict([{ severity: "critical", verdict: "covered" }], policy), - ).toBe("green"); - }); - - it("does not emit great for deterministic aggregation policy", () => { - const verdict = aggregateVerdict( - [{ severity: "critical", verdict: "covered" }], - policy, - ); - expect(verdict).not.toBe("great"); - }); - - it("counts critical blockers", () => { - expect( - criticalBlockerCount( - [ - { severity: "critical", verdict: "missing" }, - { severity: "recommended", verdict: "missing" }, - ], - policy, - ), - ).toBe(1); - }); -}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1368be0..4726496 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9,8 +9,8 @@ importers: .: devDependencies: '@agentic-workflow-kit/eval-kit': - specifier: workspace:* - version: link:packages/eval-kit + specifier: github:agentic-workflow-kit/eval-kit#v0.1.0 + version: https://codeload.github.com/agentic-workflow-kit/eval-kit/tar.gz/5a81b49553812d9a62ab49614d3b9fdaaef5d2ee ajv: specifier: ^8.17.1 version: 8.20.0 @@ -30,18 +30,14 @@ importers: specifier: ^3.2.4 version: 3.2.6(@types/debug@4.1.13)(@types/node@26.0.1)(jsdom@26.1.0)(tsx@4.22.4)(yaml@2.9.0) - packages/eval-kit: - dependencies: - ajv: - specifier: ^8.17.1 - version: 8.20.0 - devDependencies: - vitest: - specifier: ^3.2.4 - version: 3.2.6(@types/debug@4.1.13)(@types/node@26.0.1)(jsdom@26.1.0)(tsx@4.22.4)(yaml@2.9.0) - packages: + '@agentic-workflow-kit/eval-kit@https://codeload.github.com/agentic-workflow-kit/eval-kit/tar.gz/5a81b49553812d9a62ab49614d3b9fdaaef5d2ee': + resolution: {gitHosted: true, integrity: sha512-cRe37903a+SdYW5pn66G74jDiQq+T5oMXvRe5Z8OpFc2MfXF+JWgNm8EgX9giW0wYzqZAqCAvXrsip2dHEbGdg==, tarball: https://codeload.github.com/agentic-workflow-kit/eval-kit/tar.gz/5a81b49553812d9a62ab49614d3b9fdaaef5d2ee} + version: 0.1.0 + engines: {node: '>=22.13.0', pnpm: '>=11.9.0'} + hasBin: true + '@ai-sdk/gateway@3.0.140': resolution: {integrity: sha512-4VyQTHHqfZ0qI1fCcurJgYgzVDgLfV4svzBMOHGGxCu4srFkAn4ZmwFj2M0J00kNt7QLsjOtZ6PhueupeIJSjA==} engines: {node: '>=18'} @@ -4206,6 +4202,10 @@ packages: snapshots: + '@agentic-workflow-kit/eval-kit@https://codeload.github.com/agentic-workflow-kit/eval-kit/tar.gz/5a81b49553812d9a62ab49614d3b9fdaaef5d2ee': + dependencies: + ajv: 8.20.0 + '@ai-sdk/gateway@3.0.140(zod@4.4.3)': dependencies: '@ai-sdk/provider': 3.0.12 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 59e9e5c..566586f 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -1,9 +1,6 @@ # pnpm settings live here in pnpm 11: the package.json `pnpm` field is no longer read, and # `.npmrc` is auth-only. -packages: - - packages/eval-kit - # --- Supply-chain / toolchain baseline (every archetype) --- allowBuilds: esbuild: true # reviewed Vitest transitive dependency; esbuild needs its install script for the platform binary diff --git a/scripts/check_eval_static.mjs b/scripts/check_eval_static.mjs index e89bccb..736912f 100644 --- a/scripts/check_eval_static.mjs +++ b/scripts/check_eval_static.mjs @@ -63,7 +63,9 @@ try { } // Run eval-kit validate-fixtures -const evalKitBin = repoPath("packages/eval-kit/bin/eval-kit.mjs"); +const evalKitBin = repoPath( + "node_modules/@agentic-workflow-kit/eval-kit/bin/eval-kit.mjs", +); const evalConfig = repoPath("evals/eval-kit.config.json"); const result = spawnSync( process.execPath,