evals: fix false-negative regex checks with the agentic LLM judge#95440
Merged
Conversation
Contributor
Stats from current PR✅ No significant changes detected📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: 196310b |
Contributor
Tests PassedCommit: 196310b |
…t-eval 1.3 Bump @vercel/agent-eval 0.9.5 -> 1.3.0 (ships the agentic judge runtime) and adapt run-evals.js to the 1.3 CLI: `run baseline agents-md --force`, with --dry mapping to the read-only `status` command. Generated experiments now use vercel-ai-gateway/claude-code (default model claude-opus-4-8), so the OIDC token from `vc env pull` is the only credential needed — sandbox, codegen model, and judge. The judge is pinned to claude-haiku-4-5: a cheap fixed grader, identical for every run regardless of the model under test.
gaojude
force-pushed
the
jude/eval-agent-024-agentic-judge
branch
from
d2f29a5 to
a0641c6
Compare
gaojude
changed the title
agent-034-async-cookies: /await\s+cookies\(\)/ only matched the naive form and rejected the correct `await Promise.all([cookies(), headers()])`, and the no-sync-call lookbehind wrongly flagged the bare cookies() inside that array. The four await-mechanics tests collapse into one judge criterion (the promises must actually be awaited before use, in any correct form); the theme-cookie and Accept-Language content checks stay regex. agent-030-app-router-migration-hard: the home-page test rejected solutions that didn't match one exact shape (e.g. fetching extracted to a helper means no literal `fetch(` in page.tsx). Its body becomes one judge criterion; the file-existence check and the other seven tests stay deterministic. Both fixtures exclude EVAL.ts from tsconfig: the @vercel/agent-eval/eval import has no type declarations, and `next build` (run by experiments with scripts: ['build']) type-checks the fixture — same reason EVAL.tsx was already excluded.
gaojude
force-pushed
the
jude/eval-agent-024-agentic-judge
branch
from
a0641c6 to
58a13bb
Compare
Eval fixtures are sandbox workspaces with their own package.json/tsconfig, not repo code. Their EVAL.ts files may import modules that only resolve inside the sandbox (@vercel/agent-eval/eval) and are excluded from the fixture tsconfig, so type-aware linting with `project: true` fails to place them in a project. Both ignore lists (eslint.cli.config.mjs and its mirror block) stay in sync.
gaojude
force-pushed
the
jude/eval-agent-024-agentic-judge
branch
from
c4f7be1 to
196310b
Compare
aurorascharff
approved these changes
Jul 6, 2026
gaojude
added a commit
to vercel/next-evals-oss
that referenced
this pull request
Jul 7, 2026
…w pin - Bump the pinned next.js SHA to the merged canary commit for vercel/next.js#95440 (agent-034 + agent-030 converted to the agentic LLM judge; agent-040 renamed to agent-040-instant upstream). - Rerun claude-fable-5 on its outstanding evals and commit the results: agent-030 and agent-034 pass with the judge clauses executing in-sandbox; agent-040-instant fails 0/4 (classified model failure — never exported `instant`). fable-5 is fully current, so it has no ACCEPTED_STALE entry. - Reshape check-stale.mjs to (experiment, eval) pairs: ACCEPTED_STALE maps each experiment to exactly the evals it may stay stale on, so any new staleness on an accepted experiment still fails CI, and refreshing a model deletes its pairs.
gaojude
added a commit
to vercel/next-evals-oss
that referenced
this pull request
Jul 9, 2026
…w pin (#93) * Adopt agent-034/030 judge conversions; rerun claude-fable-5 at the new pin - Bump the pinned next.js SHA to the merged canary commit for vercel/next.js#95440 (agent-034 + agent-030 converted to the agentic LLM judge; agent-040 renamed to agent-040-instant upstream). - Rerun claude-fable-5 on its outstanding evals and commit the results: agent-030 and agent-034 pass with the judge clauses executing in-sandbox; agent-040-instant fails 0/4 (classified model failure — never exported `instant`). fable-5 is fully current, so it has no ACCEPTED_STALE entry. - Reshape check-stale.mjs to (experiment, eval) pairs: ACCEPTED_STALE maps each experiment to exactly the evals it may stay stale on, so any new staleness on an accepted experiment still fails CI, and refreshing a model deletes its pairs. * Add GPT 5.6 (xhigh) eval results; normalize export over the canonical eval set (#94) - gpt-5.6-xhigh + gpt-5.6-xhigh--agents-md: direct OpenAI codex, reasoningEffort=xhigh. Eval-level: 21/24 base, 22/24 with AGENTS.md. All 0% failures classified + transcript-verified as model failures. - export-results: score every experiment over the current evals/ set. Results whose eval is missing or stale (new/changed per agent-eval status --json) export as notAvailable, count against the success rate, and are excluded from flip detection and avg duration. - Restore missing MODEL_NAMES/AGENTS_MD_PAIRS entries for claude-opus-4.8, cursor-composer-2.5, gpt-5.5-pro. * Add GPT 5.6 Sol (xhigh) eval results Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base, 22/24 with AGENTS.md. All 0% evals verified as model failures at the assertion level; agent-040-instant remains unbeaten by any model. * Make Claude Fable 5's effort level explicit (high) The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed. * Add GPT 5.6 Sol (xhigh) evals; make Fable 5's effort explicit (high) (#95) * Add GPT 5.6 Sol (xhigh) eval results Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base, 22/24 with AGENTS.md. All 0% evals verified as model failures at the assertion level; agent-040-instant remains unbeaten by any model. * Make Claude Fable 5's effort level explicit (high) The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed. * Add Grok 4.5 eval results (#96) vercel-ai-gateway/opencode with the model defined via extraProviders (released 2026-07-08, not in the opencode catalogs yet; GLM 5.1 binary reused; limits from AI Gateway metadata). Eval-level: 20/24 base, 23/24 with AGENTS.md (+13, the largest docs delta measured). All 0% evals verified as model failures: <a> vs next/link (agent-000), middleware.ts vs proxy.ts (agent-031/039), missing instant export (agent-040 — still unbeaten by any model).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes false-negative regex assertions — observed rejecting correct solutions in recent eval runs — by grading those specific checks with
@vercel/agent-eval's agentic LLM judge instead. Only the assertions with demonstrated false negatives change; every deterministic check stays.agent-034-async-cookies:/await\s+cookies\s*\(\s*\)/only matched the naiveawait cookies()and rejected the correct (arguably better)await Promise.all([cookies(), headers()])— observed failing most runs of a strong model — and the no-sync-call lookbehind wrongly flagged the barecookies()inside the array. The four await-mechanics tests collapse into one judge criterion: the promises must actually be awaited before use, in any correct form. The theme-cookie and Accept-Language content checks stay regex.agent-030-app-router-migration-hard: the home-page test rejected solutions that didn't match one exact shape (e.g. data fetching extracted to a helper means no literalfetch(inpage.tsx). Its body becomes one judge criterion (async Server Component, fetches during server render, nogetServerSideProps); the file-existence check and the other seven tests stay deterministic.The judge is pinned to
claude-haiku-4-5in the generated experiment configs — a cheap fixed grader, identical for every run regardless of the model under test. Each criterion states the requirement semantically, includes one reference solution to ground the small grader, and fixes the standard of judgment (runtime correctness, not style) — no enumerated failure modes, so the judge stays flexible about equivalent forms.Supporting changes: bumps
@vercel/agent-eval0.9.5→1.3.0(ships the judge runtime) and adaptsrun-evals.jsto the 1.3 CLI (run baseline agents-md --force;--drymaps tostatus); generated experiments now run through the Vercel AI Gateway sovc env pullis the only credential setup; converted fixtures excludeEVAL.tsfrom tsconfig (the@vercel/agent-eval/evalimport has no type declarations — vercel-labs/agent-eval#166 — andnext buildtype-checks the fixture, so experiments runningscripts: ['build']would otherwise fail on it, same reasonEVAL.tsxwas already excluded). Not converted:agent-040(already redesigned upstream in #94578) andagent-041(no diagnosed false negative in the report).Verified end-to-end from this repo: converted
agent-034with codegenclaude-sonnet-4-5via the AI Gateway in a Vercel sandbox, judge pinned toclaude-haiku-4-5— passes, with the judge clause executing in-sandbox.Running it: