Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,24 @@ documented with migration notes.

## [Unreleased]

### Planned

- Additional docs for suite-specific adoption.
- Better compatibility tests for Promptfoo variable contracts.

## [0.1.6] - 2026-07-04

### Fixed

- Marked extraction-era technical-design Promptfoo follow-up docs as historical so they do not
override current two-config pointwise model-judge guidance.
- Replaced a stale release-process dependency example with a version placeholder.

### Planned
### Notes

- Additional docs for suite-specific adoption.
- Better compatibility tests for Promptfoo variable contracts.
- No runtime, schema, prompt, or CLI behavior changed.
- No npm package is published.
- Consumers may pin `github:agentic-workflow-kit/eval-kit#v0.1.6`.

## [0.1.5] - 2026-07-04

Expand Down Expand Up @@ -113,7 +121,8 @@ documented with migration notes.
- Suite-specific presets remain deferred.
- Consumer repos own their own semantics, prompts, cases, and pass/fail policies.

[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.5...main
[Unreleased]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.6...main
[0.1.6]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.5...v0.1.6
[0.1.5]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.4...v0.1.5
[0.1.4]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.3...v0.1.4
[0.1.3]: https://github.com/agentic-workflow-kit/eval-kit/compare/v0.1.2...v0.1.3
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Shared evaluation infrastructure for `agentic-workflow-kit` repositories.
```json
{
"devDependencies": {
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
}
}
```
Expand Down Expand Up @@ -70,7 +70,7 @@ Install from a Git tag in a consumer repo:
```json
{
"devDependencies": {
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
},
"scripts": {
"eval:doctor": "eval-kit doctor --config evals/eval-kit.config.json",
Expand Down Expand Up @@ -194,6 +194,7 @@ v0.1.2
v0.1.3
v0.1.4
v0.1.5
v0.1.6
v0.2.0
```

Expand Down
2 changes: 1 addition & 1 deletion docs/design/consumer-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Consumer repos should adopt eval-kit through a pinned Git tag and keep their eva
```json
{
"devDependencies": {
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
}
}
```
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/consumer-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ If you cannot state the eval goal, do not bootstrap a suite yet. Empty harnesses
```json
{
"devDependencies": {
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
}
}
```
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This guide adds a generic deterministic eval suite to a consumer repo.
```json
{
"devDependencies": {
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
}
}
```
Expand Down
18 changes: 9 additions & 9 deletions docs/reference/release-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Consumers depend on tags like:
Title:

```text
chore(release): v0.1.5
chore(release): v0.1.6
```

Required changes:
Expand Down Expand Up @@ -63,18 +63,18 @@ git checkout main
git pull --ff-only
git rev-parse HEAD

git tag -a v0.1.5 -m "v0.1.5"
git push origin v0.1.5
git tag -a v0.1.6 -m "v0.1.6"
git push origin v0.1.6
```

Verify:

```bash
git rev-parse v0.1.5^{}
git show --no-patch --decorate v0.1.5
git rev-parse v0.1.6^{}
git show --no-patch --decorate v0.1.6
```

`v0.1.5^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.5`
`v0.1.6^{}` must point to the release commit. With an annotated tag, `git rev-parse v0.1.6`
returns the tag object; `^{}` dereferences to the commit.

## GitHub Release
Expand All @@ -93,7 +93,7 @@ For each consumer repo:

```json
{
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.5"
"@agentic-workflow-kit/eval-kit": "github:agentic-workflow-kit/eval-kit#v0.1.6"
}
```

Expand All @@ -108,7 +108,7 @@ pnpm check
3. Run consumer smoke commands, for example in `technical-design`:

```bash
pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.5
pnpm eval:case -- --case case-tiny-laundry-pickup-v1 --candidate evals/cases/case-tiny-laundry-pickup-v1/reference-design.md --run-id verify-eval-kit-v0.1.6
```

4. Open a PR with dependency, lockfile, and any compatibility fixes.
Expand All @@ -120,7 +120,7 @@ Do not move the tag.
Create a new patch release:

```text
v0.1.5 -> v0.1.6
v0.1.6 -> v0.1.7
```

Then open consumer bump PRs.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Current schema:
"run_type": "deterministic",
"runner": {
"id": "generic-eval-case",
"version": "0.1.5"
"version": "0.1.6"
},
"case_ids": ["case-example-v1"],
"started_at": "2026-07-03T00:00:00.000Z",
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@agentic-workflow-kit/eval-kit",
"version": "0.1.5",
"version": "0.1.6",
"description": "Portable eval runner primitives for local eval suites.",
"private": true,
"type": "module",
Expand Down
2 changes: 1 addition & 1 deletion src/sdk.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { assertContainedPath, assertSafeId, toPosixPath } from "./paths.mjs";
const DEFAULT_SANDBOX_MODE = "read-only";
const DEFAULT_APPROVAL_POLICY = "never";
const RANDOMIZATION_METHOD = "sha256-seed-parity-v1";
const EVAL_KIT_VERSION = "0.1.5";
const EVAL_KIT_VERSION = "0.1.6";

const escapeRegExp = (value) => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");

Expand Down