fix(gateway): record partial-stream usage on client disconnect#66
Merged
Conversation
…flake The control-plane route tests use `setupAppTest` to spin up the full Hono app + memory D1 mock + admin session per test. Real work is hundreds of milliseconds, but under workspace-parallel load (workers contending for CPU, GC pauses) several of them push past vitest's 10s default and flake intermittently. 30s gives headroom without masking actual hangs.
…-stream `tokenUsageFromMessagesFrame` only returned the running usage figure on `message_stop`, leaving the per-frame observer's `state.rememberUsage` calls a no-op for the in-flight `message_start` and `message_delta` frames. When the downstream client aborted mid-stream — every Ctrl-C in the Claude Code CLI — the streaming finally block then recorded a null or stale usage, so billing telemetry under-counted every aborted session even though Anthropic already metered the output tokens against the operator's plan window. Return the running snapshot on `message_start` and `message_delta` too so `state.usage` checkpoints as the stream progresses; the finally block already records whatever was last observed when the client disconnects. Audited against sub2api (drains upstream to capture final usage even on client disconnect) — different remedy, same intent: keep recorded usage aligned with what the upstream actually metered.
Menci
force-pushed
the
precursor-streaming-improvements
branch
from
b2a3fe1 to
cce32e2
Compare
Menci
changed the title
Menci
force-pushed
the
precursor-streaming-improvements
branch
from
cce32e2 to
8ff3e0b
Compare
Menci
force-pushed
the
precursor-response-surface
branch
from
5f83545 to
1f19c40
Compare
Menci
force-pushed
the
precursor-response-surface
branch
4 times, most recently
from
c629fa7 to
dc06eca
Compare
Menci
force-pushed
the
precursor-streaming-improvements
branch
from
4a12894 to
c459978
Compare
Menci
force-pushed
the
precursor-streaming-improvements
branch
from
c459978 to
8eb6db6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tokenUsageFromMessagesFramepreviously returned the running snapshot only onmessage_stop, so the per-stream usage state never advanced past the initialmessage_start. On client disconnect mid-stream the recorded telemetry under-counted output tokens that Anthropic actually billed. Fix: snapshot on every revising frame so disconnect-time recording reflects actual partial consumption.Bumps the gateway vitest default
testTimeoutfrom 10s → 30s in the same PR — control-plane route tests usingsetupAppTest(full Hono app + memory D1 + admin session per test) flake under workspace-parallel load; real test work is hundreds of milliseconds, and 30s gives headroom without masking actual hangs.Test plan