A fair, reproducible benchmark for the different ways to give an LLM a web browser. Measured on token usage, speed, and reliability, with one identical Claude brain driving every approach.
Five locally-runnable drivers across four tool methodologies. agent-browser is run in two harnesses (a minimal API loop and the Claude Code Agent SDK) to isolate what the harness itself costs:
| Tool | Methodology | Agent loop | What it feeds the model |
|---|---|---|---|
| 🎭 Playwright MCP | accessibility tree | Claude Agent SDK (MCP) | a11y snapshot + refs |
| ⚡ agent-browser (thin) | accessibility refs | minimal API loop (Bash + CLI) | snapshot → @e1 refs |
| ⚡ agent-browser (Claude Code) | accessibility refs | Claude Agent SDK (Bash + CLI) | snapshot → @e1 refs |
| 🐍 browser-use | hybrid DOM + vision | its own Python loop | DOM elements + a screenshot |
| 🎯 Stagehand | NL primitives | its own loop (act/observe/extract) |
structured, cache-able actions |
They live at different layers, which is exactly what makes a naïve comparison misleading. This repo fixes the model and varies the browser tooling, and for agent-browser the harness around the same tool.
The diagram above is the whole idea: one Claude model, several sets of "hands," one local site.
- Playwright MCP runs on the Claude Agent SDK (the MCP server is its tool layer).
- agent-browser (thin) runs on a minimal API loop (
src/drivers/thinLoop.ts) that drives the CLI via a single Bash tool. It carries only the task prompt, a short CLI guide, and the tool schema (no Claude Code preset), so its token counts reflect the browser work rather than a large built-in system prompt re-read every turn. - agent-browser-cc runs the same agent-browser CLI on the Claude Code Agent SDK, with the same guide. It's the A/B partner to the thin driver: identical tool layer, only the harness differs, so it isolates what the heavy harness costs (and whether it buys anything).
- browser-use and Stagehand bring their own loop.
- All drivers share the same model and account: the API-loop and own-loop drivers (agent-browser thin, browser-use, Stagehand) point their Anthropic client at a tiny local proxy (
src/proxy/oauthProxy.ts); the Agent SDK drivers (Playwright MCP, agent-browser-cc) use the logged-inclaudeCLI. - Every cost is re-priced uniformly from raw token counts (each tool reports cost differently, or not at all). See
src/core/pricing.ts. - Every driver gets the same hands within the browser: UI actions only (snapshot/click/type/scroll/screenshot). Shell, in-page JS
eval, and direct network/file access are blocked, so no driver can fetch an answer instead of doing the task.
Prerequisites: Node ≥ 22, Python ≥ 3.13 + uv, Claude Code logged in (no ANTHROPIC_API_KEY needed), macOS or Linux.
npm run setup # Node + Python deps, browsers, agent-browser CLI
claude setup-token # one-time, opens browser login, prints sk-ant-oat01-...
cp .env.example .env # paste that token into .env
npm run list # see every suite, scenario, driver, model
npm run bench # run the default suiteWhat npm run setup does
npm install
npx playwright install chromium
uv venv --python 3.13 .venv && source .venv/bin/activate
uv pip install -r requirements.txt
python -m playwright install chromium
npm install -g agent-browser && agent-browser installAuthentication (all on your Claude subscription)
- Playwright MCP and agent-browser-cc authenticate via the logged-in
claudeCLI the Agent SDK spawns. - agent-browser (thin) / browser-use / Stagehand use
CLAUDE_CODE_OAUTH_TOKEN(fromclaude setup-token, in.env); the local proxy injects the Claude Code identity + beta headers so the subscription accepts the request. Verify any time withnpm run proxy:check.
npm run bench # all drivers, all scenarios, 5 reps, opus
npm run bench -- --suite=scroll --model=sonnet # one suite, on Sonnet
npm run bench -- --suite=hard --drivers=agent-browser,stagehand --reps=5| Flag | Default | Values |
|---|---|---|
--suite |
controlled |
controlled, scroll, dynamic, hard, auth, vision |
--drivers |
all five | playwright-mcp, agent-browser, agent-browser-cc, browser-use, stagehand |
--scenarios |
all in suite | scenario ids (npm run list) |
--model |
opus |
opus, sonnet, haiku, or a full model id |
--reps |
5 |
integer |
For a full run, use
scripts/clean-run.sh: it sweeps all suites and schedules the two agent-browser drivers in separate waves (they share one machine-wide daemon and can't run concurrently).
Each run lands in results/runs/<run-name>/<suite>/ (raw per-rep JSON), where <run-name> defaults to <model>_<timestamp>. Cost is normalized at the chosen model's pricing.
Each suite isolates one place where the approaches diverge. Success is checked deterministically: a scenario only reveals its gated code after the correct interaction happens, and every gated answer is generated at runtime (never a literal in the source), so an agent can't pass by guessing or by reading the benchmark's own source.
| Suite | What it tests | The question it answers |
|---|---|---|
controlled |
login, filter, gated todos | baseline: token/speed floor on easy tasks |
scroll |
below-the-fold + load-more | viewport-cap (must scroll) vs one-shot full-tree snapshot |
dynamic |
value / job that finishes after a delay | waiting / polling / re-checking behavior |
hard |
validated checkout wizard, paginated search with decoys, decoy-modal trap (all behind a cookie-consent overlay) | reliability under genuinely tricky, failure-prone flows |
auth |
login works, but a 2FA wall blocks the vault | every driver fails: fresh login can't satisfy MFA → why you need pre-baked sessions |
vision |
a code and a colour-marker rendered as server-side images (pixels only) | isolates vision-capable drivers from DOM-only ones |
Every fixture is a small React app served from memory (esbuild-bundled, fully offline). Gated answer codes are generated at runtime and released only after the genuine interaction, behind session-gated endpoints (cookies, per-session refs, sequential paging) or, for vision, rendered into pixels, so they can't be scraped from the DOM, the JS bundle, or a direct API call.
Every run streams per-rep progress to the console and writes raw per-rep JSON to results/runs/<run-name>/<suite>/: one file per rep, plus a raw.json with all results and per-driver aggregates.
Regenerate the comparison plots for any run with:
python py/plots.py # latest run (PNGs land in <run>/analysis/plots/)
python py/plots.py --run <name> # a specific runThe numbers below are one full run on claude-sonnet-4-6 (5 reps per scenario, ~300 tasks). Pass rates are a near tie (95 to 100 percent for every driver), so the interesting differences are in how much context each approach carries.
Tokens per browser step (median + IQR). The same agent-browser CLI costs 2.7k tokens per step on a minimal API loop and 35k inside Claude Code; Playwright MCP carries the full accessibility snapshot plus tool schemas on every call:
Same CLI, two harnesses (tokens per task, median + IQR). Identical tool layer and commands; only the harness differs:
Per suite (median tokens per task, log scale). The gap is task-shaped: huge on extraction-style suites (controlled, dynamic, vision), narrower where page content dominates everyone's context (scroll, hard):
Cost note: tokens, not dollars. Most of the heavy drivers' volume is cache reads, which are nearly free on a subscription, so the billed gap is much smaller than the token gap.
| Metric | Meaning |
|---|---|
| Reliability | success rate over N reps (the least-confounded signal) |
| Tool calls / turns | browser actions / LLM round-trips; fewer = more batching |
| Duration | wall-clock per task (median & p95) |
| Tokens | fresh input + cache reads + output the model processed |
| Cost | normalized from tokens at the run's model pricing (a uniform cross-tool comparison, not your subscription bill) |
Add a suite: create src/suites/<name>/{site.ts, scenarios.ts, index.ts} and register it in src/suites/registry.ts.
Add a driver: implement the Driver interface (src/core/types.ts) and register it in src/drivers/index.ts. Tool layers can reuse makeAgentSdkDriver (Agent SDK) or runThinLoop (minimal API loop).
This benchmark measures these projects; all credit for the tools themselves goes to their authors:
- Playwright MCP by Microsoft
- agent-browser by Vercel Labs
- browser-use by Browser Use
- Stagehand by Browserbase
- Claude Agent SDK by Anthropic
MIT © 2026 Nicola Sosio