Replay · Evaluate · Compare · Assert · Catch Regressions — in CI
Quick Start · Why · DSL · Ecosystem · Examples · vs Others · Documentation · Releases
npm install -g @agentbench/cli
agentbench init --quick30 秒。从安装到看到测试通过。 然后你就可以把测试文件改成你自己的 Agent。
import { agentbench } from '@agentbench/core'
const suite = agentbench.suite('My first agent test')
suite.test('handles refund requests correctly', async ({ agent, expect }) => {
const response = await agent.toolCall('refund', { orderId: 'ORD-123', reason: 'defective' })
expect(response).tool('refund').toBeCalledWith({ orderId: 'ORD-123' })
expect(response).output.semanticMatch('refund initiated')
})跑一次：
agentbench test看到结果：
PASS tests/my-agent.test.ts
✓ handles refund requests correctly (1.2s)
✓ responds politely to greetings (0.8s)
✓ escalates when unable to help (1.5s)
Tests: 3 passed, 3 total
Time: 3.5s
- 把你的 Agent 代码放到
src/agent.ts - 在
tests/里写断言 —— 22 种 matcher 任选 agentbench test跑起来- 加一个 GitHub Actions workflow，PR 里自动拦住回归
从 0 到 CI 门禁，5 分钟。
Testing AI agents is fundamentally different from testing deterministic software. AgentBench is built on a layered strategy: deterministic assertions (tools, tokens, latency) run on every PR; LLM-assisted quality scores (correctness, safety, faithfulness) gate pre-release. Read the Agent Testing Pyramid for the full strategy, and the Anti-Patterns to avoid the most common testing mistakes.
"I tracked my time. Coding was 10%. Testing was 90%. Not because I'm slow — because there was no tool."
AI Agents are unpredictable. A prompt tweak, a model upgrade, or a tool swap can silently degrade your agent -- and most teams discover this only when users complain.
AgentBench gives you the same testing rigor for your AI agents that you expect for your software.
|
|
|
await expect(runResult)
.tool("search").toBeCalled()
.tool("search").toBeCalledWith({ query: "refund policy" })
.output().toContain("30 days")
.tokens().toBeLessThan(4096)
.latency().toBeLessThan(5000)
.score("correctness").toBeGreaterThan(7)
.run()
|
The most fluent way to test an AI agent. Chainable, type-safe, reads like English.
import { expect } from '@agentbench/core'
const result = await expect(runResult)
.status().toBeCompleted() // Agent finished successfully
.tool("search_docs").toBeCalled() // Called the right tool
.tool("search_docs").toBeCalledWith({ // Called with correct args
query: "refund policy"
})
.tool("hallucinate").not.toBeCalled() // No forbidden tools
.output().toContain("30 days") // Output has correct info
.output().toMatchRegex(/refund.*policy/i) // Pattern validation
.tokens().toBeLessThan(4096) // Token budget respected
.latency().toBeLessThan(5000) // Under 5 seconds
.score("correctness").toBeGreaterThan(7) // Quality threshold met
.score("safety").toBeGreaterThan(8) // Safety threshold met
.run()
if (!result.allPassed) process.exit(1)All 22 Matchers
| Category | Matchers |
|---|---|
| Tool | toBeCalled(), toBeCalledWith(), toBeCalledTimes(), not.toBeCalled() |
| Tokens | toBeLessThan(), toBeGreaterThan(), toBeBetween() |
| Latency | toBeLessThan(), toBeGreaterThan(), firstToken().toBeLessThan() |
| Output | toContain(), not.toContain(), toEqual(), toMatchRegex(), toMatchSchema(), toMatchSnapshot() |
| Score | toBeGreaterThan(), toBeLessThan(), toBeBetween() |
| Status | toBeCompleted(), toBe("passed") |
| Compound | all(), any() |
| Provider | Package | Capabilities | Status |
|---|---|---|---|
| OpenAI | @agentbench/openai |
Streaming, reasoning, tool-calling, vision, function-calling, JSON mode | GA |
| Anthropic | @agentbench/anthropic |
Streaming, reasoning, tool-calling, vision | GA |
| Gemini | @agentbench/gemini |
Streaming, embeddings, tool-calling, vision, JSON mode | GA |
| DeepSeek | @agentbench/deepseek |
Streaming, reasoning, tool-calling, JSON mode | GA |
| Azure OpenAI | @agentbench/azure-openai |
Streaming, embeddings, tool-calling, vision | Beta |
| OpenRouter | @agentbench/openrouter |
Streaming, tool-calling (pass-through to 200+ models) | Beta |
| Groq | @agentbench/groq |
Streaming, tool-calling, JSON mode (fast inference) | Beta |
| Mistral | @agentbench/mistral |
Streaming, embeddings, tool-calling, JSON mode | Beta |
| Cohere | @agentbench/cohere |
Streaming, embeddings, tool-calling | Beta |
| Ollama | @agentbench/ollama |
Streaming, embeddings, tool-calling, JSON mode (local) | Beta |
| vLLM | @agentbench/vllm |
Streaming, tool-calling (OpenAI-compatible) | Beta |
| LM Studio | @agentbench/lm-studio |
Streaming, tool-calling (OpenAI-compatible, local) | Beta |
| Package | Description | Status |
|---|---|---|
@agentbench/core |
Core engine -- Runner, Tracer, Evaluator, Assertion DSL, Replay, Diff, Coverage | GA |
@agentbench/mcp |
MCP client wrapper for tool calls and resource access | GA |
@agentbench/adapter |
Generic adapter for LangGraph, CrewAI, LlamaIndex, and custom agents | GA |
@agentbench/langgraph |
First-class LangGraph adapter with state graph tracing | GA |
@agentbench/provider-utils |
Shared base classes for building custom providers | GA |
agentbench (Python) |
Full Python SDK -- Runner, Tracer, Assertions, HTTP client | GA |
14 production-quality reference implementations. Each demonstrates how to test a specific kind of AI agent with comprehensive test suites.
| # | Example | Category | What It Demonstrates |
|---|---|---|---|
| 1 | Hello Agent | General | Minimal starter -- the template generated by agentbench init |
| 2 | Customer Support | customer-support | Multi-turn support agent with RAG, tool calling, escalation, regression suite |
| 3 | Research Agent | research | Multi-step research with web search, source verification, citation accuracy |
| 4 | RAG Agent | rag | Retrieval-augmented generation with grounding, context-window, latency tests |
| 5 | SQL Agent | sql | Text-to-SQL with schema awareness, join/aggregation, SQL injection prevention |
| 6 | Code Review Agent | coding | Security review, code quality, false-positive detection, large diff handling |
| 7 | Coding Agent | coding | Code generation, bug-fix loop, refactoring, test-driven development |
| 8 | Tool-Calling Agent | tool-calling | Complex tool orchestration -- selection, parallel calls, ordering, error handling |
| 9 | MCP Agent | mcp | Model Context Protocol -- tool discovery, resource access, multi-server, lifecycle |
| 10 | LangGraph Agent | agent-workflow | State graph testing -- workflow paths, transitions, conditional edges, human-in-loop |
| 11 | OpenAI Agent SDK | agent-workflow | Guardrails, handoffs, tool use, tracing integration |
| 12 | CrewAI Agent | multi-agent | Multi-agent crews -- task completion, delegation, sequential workflow, output quality |
| 13 | LlamaIndex Agent | rag | Query engine, chat engine, tool integration, index quality |
| 14 | Multi-Agent Workflow | multi-agent | Complex orchestration -- handoff, consensus, concurrency, failure recovery |
Each example includes:
- README with quick start, architecture diagram, and key takeaways
- 3-5 test suites with 8+ test cases
- At least 3 different assertion types (tool, output, score, latency, tokens)
- Replay test suite (zero-cost, deterministic)
- CI workflow file (
.github/workflows/agentbench.yml) - Dataset of 20+ test inputs
- 100% pass rate when run against their target agent
# Run any example
cd examples/customer-support-agent
cp .env.example .env # Add your API key
npm install
agentbench testAgentBench is not an observability platform or an evaluation library. It is a testing framework -- the same category as Jest, Playwright, and Pytest, but purpose-built for the non-deterministic world of AI agents.
| Tool | Category | Best For |
|---|---|---|
| AgentBench | Agent Testing | CI/CD regression testing, assertions, replay |
| LangSmith | Observability | Debugging traces, monitoring production |
| DeepEval | LLM Evaluation | Evaluating output quality metrics |
| Promptfoo | Prompt Testing | Comparing prompt variants |
| Playwright | Browser Testing | Testing browser interactions |
| Jest | Unit Testing | Testing deterministic code |
| Tool | Category | What It Does | Key Difference from AgentBench |
|---|---|---|---|
| LangSmith | LLM Observability | Trace, monitor, and annotate LLM calls | LangSmith helps you observe what happened. AgentBench helps you assert what should happen and gate on it in CI. Use LangSmith to debug; use AgentBench to block broken agents from shipping. |
| DeepEval | LLM Evaluation | Metrics and benchmarks for LLM outputs | DeepEval evaluates output text. AgentBench tests the entire agent -- which tools were called, in what order, with what arguments, token budgets, latency budgets, and whether quality regressed from the last run. |
| Promptfoo | Prompt Testing | Compare prompt variants | Promptfoo tests prompts. AgentBench tests agents -- prompts + tools + chains + state + multi-turn conversation. |
| OpenAI Evals | LLM Benchmarking | Standardized eval suites for models | OpenAI Evals is a benchmark runner. AgentBench is a developer testing workflow -- assertions, replay, regression detection, CI/CD, VS Code integration. |
| Jest / Vitest | Unit Testing | JavaScript/TypeScript test runner | Jest expects deterministic outputs. AI agents are non-deterministic. AgentBench provides LLM-aware assertions, replay, and regression detection that Jest cannot. |
| Playwright | Browser Testing | Automate and test browser interactions | Playwright tests browser apps. AgentBench tests AI agents. Complementary -- an agent that uses a browser can be tested with both. |
Why not just use LangSmith? LangSmith is for observability -- seeing what happened. AgentBench is for testing -- asserting what should happen and gating on it. You use LangSmith to debug; you use AgentBench in CI to prevent broken agents from reaching production.
Why not just use Jest?
Jest expects deterministic outputs. AI agents are non-deterministic. AgentBench provides LLM-aware assertions (tool().toBeCalled(), score("correctness").toBeGreaterThan(7)), replay, and regression detection that Jest cannot.
Why not just use DeepEval? DeepEval evaluates the output text. AgentBench tests the agent behavior -- which tools were called, in what order, with what arguments, how many tokens were used, whether the response was fast enough, and whether quality regressed from the last run.
| Metric | Value | Metric | Value | |
|---|---|---|---|---|
| TypeScript Files | 120+ | CLI Commands | 12 | |
| Lines of Code | 22,000+ | Providers Supported | 12+ | |
| Packages | 8 | Official Examples | 14 | |
| API Endpoints | 18 | VS Code Extension | Published | |
| Unit Tests | 391+ | Documentation Pages | 25+ | |
| TS Errors | 0 | npm Package | Published |
| Phase | Milestone | |
|---|---|---|
| M0-M3 | Foundation, Core Engine, Evaluation, Regression & Replay | GA |
| M4-M7 | Experiments & Coverage, SDK Ecosystem, Platform, Polish | GA |
| v0.3.0 | Brand refresh, 14 examples, 12+ providers, dataset system, GitHub integration, VS Code extension, benchmark marketplace, documentation site | Current |
| v0.4.0 | Ecosystem -- GitHub Actions PR integration, full dataset system, VS Code Trace Viewer, benchmark validation pipeline | Q4 2026 |
| v0.5.0 | Enterprise -- Team workspaces, AgentBench Cloud, SSO, audit logs | Q1 2027 |
| v1.0.0 | Standard -- Plugin marketplace, stable v1 API, Python SDK v1.0, certification program | Q3 2027 |
agentbench/
├── apps/
│ ├── web/ Next.js 15 Dashboard + REST API + Benchmark Marketplace
│ ├── cli/ Commander.js CLI (12 commands)
│ └── docs/ VitePress documentation site (25+ pages)
├── packages/
│ ├── core/ @agentbench/core -- Engine
│ │ ├── runner/ Agent Runner
│ │ ├── tracer/ Execution Tracer + LLM Interceptors
│ │ ├── evaluator/ Rule + LLM + Hybrid Judge
│ │ ├── assertion/ Chained Assertion DSL
│ │ ├── snapshot/ Snapshot Manager
│ │ ├── replay/ Replay Engine
│ │ ├── diff/ Diff Engine
│ │ ├── experiment/ A/B Testing Engine
│ │ ├── coverage/ Coverage Analysis
│ │ ├── reporter/ Report Generator (JSON/MD/HTML/JUnit)
│ │ ├── dataset/ Dataset Management (CSV/JSON/JSONL)
│ │ ├── storage/ Storage Abstraction (Postgres + Memory)
│ │ ├── types/ TypeScript Type Definitions
│ │ └── utils/ Token Counter + JSON Validator
│ ├── openai/ @agentbench/openai
│ ├── anthropic/ @agentbench/anthropic
│ ├── gemini/ @agentbench/gemini
│ ├── deepseek/ @agentbench/deepseek
│ ├── azure-openai/ @agentbench/azure-openai
│ ├── openrouter/ @agentbench/openrouter
│ ├── groq/ @agentbench/groq
│ ├── mistral/ @agentbench/mistral
│ ├── cohere/ @agentbench/cohere
│ ├── ollama/ @agentbench/ollama
│ ├── vllm/ @agentbench/vllm
│ ├── lm-studio/ @agentbench/lm-studio
│ ├── provider-utils/ @agentbench/provider-utils
│ ├── mcp/ @agentbench/mcp
│ ├── adapter/ @agentbench/adapter
│ ├── langgraph/ @agentbench/langgraph
│ └── typescript-config/ Shared TSConfig
├── examples/ 14 official examples
├── sdk-python/ Python SDK
├── vscode-extension/ VS Code extension
├── benchmark-registry/ Benchmark registry + DB
├── docs/ Internal docs (architecture, schema, roadmap)
├── docker-compose.yml PostgreSQL 16 + Redis 7
└── .github/workflows/ CI pipeline
| Document | |
|---|---|
| Documentation Hub | Full docs site built with VitePress |
| Web Dashboard Guide | New user walkthrough -- setup, test runs, and reading results |
| Getting Started | Step-by-step tutorial |
| API Reference | 18 endpoints with curl examples |
| CLI Reference | 12 commands with all options |
| SDK Guide | Usage for OpenAI, Anthropic, MCP, Adapter, and more |
| Architecture | System design and data flow |
| Roadmap | v0.3 through v1.0 plans |
| Deployment | Docker, Vercel, self-hosted |
| FAQ | 20+ common questions |
| Glossary | 50+ terminology definitions |
| Testing Pyramid | Layered testing strategy for AI agents |
| Anti-Patterns | 8 common testing mistakes and how to avoid them |
| Ecosystem Integrations | Claude Code, LangChain/CrewAI, Vercel AI SDK |
| GitHub Action | One-step CI integration for agent regression testing |
AgentBench is open source and community-driven. We welcome all contributions.
| Link | |
|---|---|
| Discussions | github.com/1304674612/agentbench/discussions |
| Issue Tracker | github.com/1304674612/agentbench/issues |
| Contributing Guide | CONTRIBUTING.md |
| Security Policy | SECURITY.md |
| Code of Conduct | CODE_OF_CONDUCT.md |
Built with love for the AI agent community
Apache 2.0 License - (c) 2026 AgentBench Contributors
If AgentBench saves you from shipping a broken agent, give it a star