fix(agents-server): release pull-wake claim row even when in-memory token is missing

[kevin-dp]

Summary

Fixes #4340 — pull-wake claims leaking in consumer_claims when the in-memory ClaimWriteTokenStore no longer holds the consumer's token at the time sendDone arrives.

The release path in callback-forward was gated by stillOwnsClaim, an in-memory check. When that check fails — server restart between mint and done, parallel wakes evicting each other's tokens, or a retry after a transient updateStatus failure — the entire release block is skipped: materializeReleasedClaim never runs, the entity stays stuck at status='running', and the consumer_claims row stays active indefinitely.

The steady-state "send one message, wait" path is not affected by this bug: it releases correctly via the runtime's sendDone after the idleTimeout window (default 5 min via packages/agents/src/bootstrap.ts:146). The bug only fires in the failure modes documented in Test scenarios below.

Root cause

packages/agents-server/src/routing/internal-router.ts had all three release actions behind the same in-memory gate:

if (entity && stillOwnsClaim) {
  await materializeReleasedClaim(...)   // DB row release
  await updateStatus(entity.url, 'idle') // entity status
  clearStream(...)                       // in-memory token cleanup
  await onEntityChanged(entity.url)
} else if (stillOwnsClaim) {
  clearStream(...)
} else if (entity) {
  log.info('done ignored for stale claim ...')
}

stillOwnsClaim is the right gate for write authorization during the agent run, but it's the wrong gate for releasing the DB row, which is keyed by (consumerId, epoch). The DB primary key is authoritative identity — the in-memory token state is orthogonal.

The fix

Three concerns, three gates:

1. DB row release (materializeReleasedClaim) — runs whenever epoch is defined. (consumerId, epoch) is the DB primary key; that's enough.
2. Entity status → idle + onEntityChanged — runs when entityCleared || stillOwnsClaim. entityCleared is a new return field from materializeReleasedClaim, set to true only when our (consumerId, epoch) was the active dispatch row and we just nulled it out. The || handles two non-trivial cases: server restart (DB has us active, token is gone) and retry after a failed updateStatus (state cleared on first attempt, token still held).
3. In-memory token cleanup (clearStream) — remains gated by stillOwnsClaim so a newer consumer's token is never cleared out from under it.

materializeReleasedClaim API change

- ): Promise<ConsumerClaim | null> {
+ ): Promise<{ claim: ConsumerClaim | null; entityCleared: boolean }> {

Only one production caller (internal-router.ts); both that caller and the test mock are updated. The .returning() on the entityDispatchState UPDATE now reports whether our row was actually cleared (vs. a no-op because a newer claim is active).

Test scenarios

The fix decouples the three concerns. Below: ✓ = action happens, × = action does NOT happen (and is correct that it doesn't).

Scenario	entityCleared	stillOwnsClaim	DB row released	Entity → idle
A. Happy path (mint + done)	true	true	✓ released	✓ goes idle
B. Server restart (no in-memory token, DB row still active)	true	false	✓ released	✓ goes idle
C. Newer wake (wake-1 done after wake-2 takes over the stream)	false	false	✓ wake-1's row released	× stays running — wake-2 is in flight
D. Retry (first done's updateStatus threw; same done retried)	false	true	✓ no-op (already released)	✓ goes idle
E. Legacy stale-done test (test setup never materialized active claim; token evicted)	false	false	no row to release	× stays running — newer claim conceptually in flight

New tests in packages/agents-server/test/webhook-forward-routing.test.ts > claim release on done callback (regression for #4340) cover scenarios A–C. Existing tests in server-claim-write-token.test.ts cover D and E and continue to pass after the fix.

Verified

• Unit tests: deterministic. Pre-fix, B and C fail (zero invocations of materializeReleasedClaim); D fails (updateStatus skipped on retry). Post-fix, all five scenarios produce the documented behavior.
• Manual run-through against a local desktop + agents-server: send one message, dispatch claims (active_count: 0 → 1), agent completes, runtime calls sendDone after idleTimeout, server fires the new release path, active_count: 1 → 0, entity status transitions back to idle.

Not addressed in this PR

• Pre-existing orphan rows: rows that already leaked from prior runs of the unfixed code can't be released because no fresh done callback is coming. Would need a reaper job or admin command.
• lease_expires_at: null issue (Pull-wake: heartbeat path nulls out lease_expires_at on consumer_claims #4341): independent. Without a lease, even a reaper job can't time-out claims safely.

Base branch note

This PR targets fix-pull-wake (#4339), not main, because materializeReleasedClaim was introduced in #4308 which is part of the fix-pull-wake lineage but not yet in main. Merge order: this → fix-pull-wake → main.

🤖 Generated with Claude Code

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
223	1	222	2

View the full list of 1 ❄️ flaky test(s)

test/horton-pull-wake-e2e.test.ts > pull-wake Horton e2e with mocked LLM > dispatches explicit runner-policy wakes and Horton writes mocked responses

Flake rate in main: 100.00% (Passed 0 times, Failed 8 times)

Stack Traces | 0.0775s run time

AssertionError: expected 500 to be 204 // Object.is equality

- Expected
+ Received

- 204
+ 500

 ❯ test/horton-pull-wake-e2e.test.ts:183:28

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}