test(agent): live smoke through real gateway to real Anthropic#11
Merged
Conversation
Mirrors the gateway's `test:smoke` pattern for the agent binaries: an #[ignore]'d test (tests/smoke.rs) that boots the real beyond-ai gateway with the caller's ANTHROPIC_API_KEY as the managed pool key + the dev signing key, then drives the real beyond-ai-agent binary — holding only a bai_v1 virtual key — through it to REAL Anthropic, performing a live tool round-trip (read a file, echo its token). This validates the dialect decoder against a real provider's SSE (real tool_use streaming deltas), through real gateway auth + key-swap — closing the gap that the mock-fixture e2e couldn't: nothing else here talks to a real model. Skips cleanly when ANTHROPIC_API_KEY is unset (so CI stays hermetic). Also: hoist gateway_bin() + the dev key constants into tests/common so the mock gateway_e2e and the live smoke share them. New mise task: test:smoke:agent (auto-loads .env, like test:smoke). Verified live: 2 steps, real `[tool: read]` round-trip, model returned the exact token. No secrets committed (key read from env / gitignored .env). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Mirrors the gateway's
test:smokepattern for the agent binaries: an#[ignore]d test (crates/agent/tests/smoke.rs) that boots the realbeyond-aigateway with the caller'sANTHROPIC_API_KEYas the managed pool key (+ the dev signing key), then drives the realbeyond-ai-agentbinary — holding only abai_v1virtual key — through it to real Anthropic, performing a live tool round-trip (read a file, echo its token).Why
This is the one thing the mock-fixture e2e couldn't prove: the dialect decoder against a real provider's SSE (real
tool_usestreaming deltas), through real gateway auth + key-swap. Nothing else in the suite talks to a real model.Skips cleanly when
ANTHROPIC_API_KEYis unset, so CI stays hermetic (this test never runs in CI).Verified live
Real claude-haiku-4-5 called the
readtool through the gateway, the loop ran it on a real file, fed the content back, and the model echoed the exact token.Also
gateway_bin()+ the dev-key constants intotests/commonso the mockgateway_e2eand the live smoke share them.test:smoke:agent(auto-loads.env, liketest:smoke)..env; only placeholdersk-ant-…in docs).🤖 Generated with Claude Code