Skip to content

Commit a063d52

Browse files
amitpaz1claude
amitpaz1
and
claude
committed
feat: implement 10-feature product roadmap (v0.6.0)
Phase 1 — Foundation/MVP: - Feature-1: Deterministic run profiles with seed, sampling, grader defaults (profiles.py, --profile CLI option) - Feature-4: Strict mypy typing enforcement in CI pipeline - Feature-5: Health check command (agenteval doctor) - Feature-6: Interactive suite scaffolding (agenteval init) and YAML linting (agenteval lint) Phase 2 — Enhancement: - Feature-2: Regression gate policies with YAML-declared thresholds (gates.py, --gate CLI option on compare) - Feature-3: Unified run report artifacts in JSON and Markdown (reports.py, agenteval report command, --report on run) - Feature-10: Cross-language onboarding kits (TypeScript agent, GitHub Actions templates, Docker examples) Phase 3 — Scale/Growth: - Feature-7: Distributed execution reliability (dead-letter queue, task status tracking, run resumption, worker diagnostics) - Feature-8: Historical trend analysis with budget guardrails (trends.py, agenteval trends command) Phase 4 — Future/Vision: - Feature-9: Optional local web dashboard with runs, detail, and trends views (stdlib http.server, Chart.js, vanilla JS) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 65ea2f9 commit a063d52

39 files changed

Lines changed: 1993 additions & 20 deletions

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,6 @@ jobs:
2222
run: pip install pip-audit && pip-audit --desc
2323
continue-on-error: true
2424
- run: pytest
25+
- name: Type check
26+
run: pip install mypy types-PyYAML && mypy src/agenteval/ --ignore-missing-imports
27+
continue-on-error: true

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,13 @@ __pycache__/
55
dist/
66
build/
77
*.pyc
8+
9+
# Aperant data directory
10+
.auto-claude/
11+
12+
# Claude Code / BMAD
13+
.claude/
14+
_bmad/
15+
16+
# SQLite databases
17+
*.db

‎examples/docker/Dockerfile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM python:3.12-slim
2+
3+
WORKDIR /work
4+
5+
# Install agenteval with distributed extras for optional Redis support
6+
RUN pip install --no-cache-dir agentevalkit[distributed]
7+
8+
# Copy example suite (override at runtime with -v)
9+
COPY suite.yaml .
10+
11+
CMD ["agenteval", "run", "--suite", "suite.yaml"]

‎examples/docker/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Running AgentEval in Docker
2+
3+
This example shows how to run agenteval in a Docker container, with an
4+
optional Redis service for distributed mode.
5+
6+
## Quick start
7+
8+
```bash
9+
# Build the image
10+
docker build -t agenteval .
11+
12+
# Run a suite
13+
docker run --rm -v $(pwd)/suite.yaml:/work/suite.yaml agenteval \
14+
agenteval run --suite suite.yaml --agent my_agent:run
15+
```
16+
17+
## With Docker Compose (distributed mode)
18+
19+
```bash
20+
docker compose up
21+
```
22+
23+
This starts a Redis instance and runs the agenteval worker. You can then
24+
submit jobs from the agenteval container.
25+
26+
## Customisation
27+
28+
- Mount your agent code into `/work` to make it importable.
29+
- Set `OPENAI_API_KEY` via environment variable or `.env` file.
30+
- Add extra pip packages in the Dockerfile as needed.

‎examples/docker/docker-compose.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
services:
2+
redis:
3+
image: redis:7-alpine
4+
ports:
5+
- "6379:6379"
6+
7+
agenteval:
8+
build: .
9+
depends_on:
10+
- redis
11+
environment:
12+
- AGENTEVAL_REDIS_URL=redis://redis:6379/0
13+
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
14+
volumes:
15+
- .:/work
16+
command: ["agenteval", "run", "--suite", "suite.yaml"]
17+
18+
worker:
19+
build: .
20+
depends_on:
21+
- redis
22+
environment:
23+
- AGENTEVAL_REDIS_URL=redis://redis:6379/0
24+
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
25+
volumes:
26+
- .:/work
27+
command: ["agenteval", "worker", "--redis-url", "redis://redis:6379/0"]

‎examples/docker/suite.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: docker-example-tests
2+
agent: my_agent:run
3+
4+
cases:
5+
- name: basic-response
6+
input: "Hello, agent!"
7+
expected:
8+
contains: "Hello"
9+
grader: contains
10+
11+
- name: factual-check
12+
input: "What is 2 + 2?"
13+
expected:
14+
contains: "4"
15+
grader: contains

‎examples/github-actions/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# GitHub Actions CI Templates for AgentEval
2+
3+
Reusable workflow templates for running agenteval in CI.
4+
5+
## Templates
6+
7+
### basic.yml
8+
9+
Minimal workflow: installs agenteval, runs a test suite, and fails on
10+
non-zero exit code.
11+
12+
### with-comparison.yml
13+
14+
Runs a suite, compares results with a stored baseline, and posts a
15+
summary comment on the pull request.
16+
17+
### with-gates.yml
18+
19+
Runs a suite with quality gates. Fails the build if any metric
20+
regresses beyond the configured threshold.
21+
22+
## Usage
23+
24+
Copy the desired `.yml` file into your repository's `.github/workflows/`
25+
directory and adjust the suite path, agent reference, and any thresholds
26+
to match your project.

‎examples/github-actions/basic.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Basic agenteval CI workflow.
2+
# Runs a test suite and fails if any case fails.
3+
4+
name: AgentEval Basic
5+
on:
6+
push:
7+
branches: [main]
8+
pull_request:
9+
branches: [main]
10+
11+
jobs:
12+
eval:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.12"
20+
21+
- name: Install agenteval
22+
run: pip install agentevalkit
23+
24+
- name: Run evaluation suite
25+
run: agenteval run --suite suite.yaml --agent my_agent:run
26+
27+
- name: Upload results
28+
if: always()
29+
uses: actions/upload-artifact@v4
30+
with:
31+
name: eval-results
32+
path: agenteval.db

‎examples/github-actions/with-comparison.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# AgentEval CI workflow with baseline comparison and PR comment.
2+
3+
name: AgentEval Compare
4+
on:
5+
pull_request:
6+
branches: [main]
7+
8+
permissions:
9+
pull-requests: write
10+
11+
jobs:
12+
eval:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.12"
20+
21+
- name: Install agenteval
22+
run: pip install agentevalkit
23+
24+
- name: Run evaluation suite
25+
run: agenteval run --suite suite.yaml --agent my_agent:run --format json -o results.json
26+
27+
- name: Compare with baseline
28+
run: agenteval compare --baseline baseline.json --current results.json --format markdown -o comparison.md
29+
30+
- name: Post PR comment
31+
if: github.event_name == 'pull_request'
32+
env:
33+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
34+
run: |
35+
gh pr comment ${{ github.event.number }} \
36+
--body-file comparison.md \
37+
--edit-last || \
38+
gh pr comment ${{ github.event.number }} \
39+
--body-file comparison.md
40+
41+
- name: Upload artifacts
42+
if: always()
43+
uses: actions/upload-artifact@v4
44+
with:
45+
name: eval-artifacts
46+
path: |
47+
results.json
48+
comparison.md

‎examples/github-actions/with-gates.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# AgentEval CI workflow with quality gates.
2+
# Fails the build if metrics regress beyond thresholds.
3+
4+
name: AgentEval Gates
5+
on:
6+
pull_request:
7+
branches: [main]
8+
9+
permissions:
10+
pull-requests: write
11+
12+
jobs:
13+
eval:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- uses: actions/setup-python@v5
19+
with:
20+
python-version: "3.12"
21+
22+
- name: Install agenteval
23+
run: pip install agentevalkit
24+
25+
- name: Run evaluation suite
26+
run: agenteval run --suite suite.yaml --agent my_agent:run --format json -o results.json
27+
28+
- name: Compare with gates
29+
run: |
30+
agenteval compare \
31+
--baseline baseline.json \
32+
--current results.json \
33+
--gate pass_rate:0.95 \
34+
--gate avg_score:0.8 \
35+
--gate max_latency_ms:5000 \
36+
--format json -o gate-results.json
37+
38+
- name: Check gate status
39+
run: |
40+
python3 -c "
41+
import json, sys
42+
data = json.load(open('gate-results.json'))
43+
if not data.get('gates_passed', False):
44+
for g in data.get('failures', []):
45+
print(f\"GATE FAILED: {g['gate']} — got {g['actual']}, required {g['threshold']}\")
46+
sys.exit(1)
47+
print('All quality gates passed.')
48+
"
49+
50+
- name: Post PR comment
51+
if: always() && github.event_name == 'pull_request'
52+
env:
53+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
54+
run: |
55+
STATUS=$(python3 -c "import json; d=json.load(open('gate-results.json')); print('passed' if d.get('gates_passed') else 'FAILED')")
56+
gh pr comment ${{ github.event.number }} \
57+
--body "## AgentEval Gate Results: ${STATUS}
58+
$(cat gate-results.json | python3 -m json.tool)" \
59+
--edit-last || \
60+
gh pr comment ${{ github.event.number }} \
61+
--body "## AgentEval Gate Results: ${STATUS}
62+
$(cat gate-results.json | python3 -m json.tool)"

0 commit comments

Comments
 (0)