fix(voice): preserve completed tool results in chat context on interruption#6349
fix(voice): preserve completed tool results in chat context on interruption#6349hello-rory wants to merge 9 commits into
Conversation
|
|
…uption Completed FunctionCall/FunctionCallOutput pairs were dropped when the speech carrying them was interrupted, so the next LLM inference had no record the tool ran and re-issued the same call, duplicating side effects for non-idempotent tools (livekit#3702). - commit _previous_tools_messages at the start of the pipeline reply turn, before any interruption gate, instead of after playout - on the post-playout interruption return, commit completed results from tool_output.output (pipeline and realtime paths, per livekit#4877); in realtime, also push the outputs to the realtime session so its server-side state doesn't retain a dangling function call - exclude agent handoff tools from the interrupted-path commit: the handoff is not applied on an interrupted speech, and recording it as completed would prevent the LLM from retrying it
hello-rory
force-pushed
the
fix/preserve-tool-results-on-interrupt
branch
from
871b51f to
f1c1d35
Compare
hello-rory
changed the title
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| chat_ctx = self._rt_session.chat_ctx.copy() | ||
| chat_ctx.items.extend(interrupted_fnc_outputs) | ||
| try: | ||
| await self._rt_session.update_chat_ctx(chat_ctx) |
There was a problem hiding this comment.
This push is unconditional, but its effect depends on auto_tool_reply_generation:
- OpenAI (
False): just records the result — clean fix. - Gemini (
True): sending the function response is the trigger to generate, so we'd start a spoken tool reply right after the user interrupted.
Suggest gating: keep the local _upsert_item, but only update_chat_ctx when not capabilities.auto_tool_reply_generation. Note the local upsert alone doesn't prevent re-execution (realtime generation is driven by server-side state), so maybe log a warning in that case?
There was a problem hiding this comment.
Hmm... I was digging into this a bit. I am less confident in the realtime code path changes here, so I just dropped them from the diff. Our issue was specifically with pipeline so I don't want to get out of scope. I was drilling Fable about it, and it came back at me with this:
The one finding that actually matters: the Gemini "local-only" outputs aren't local-only forever
The gate holds interrupted outputs back from the server, but they still go into self._agent._chat_ctx. Google's update_chat_ctx (realtime_api.py:614) works by diffing the incoming context against the plugin's own _chat_ctx — and converts every new function_call_output in the diff into a LiveClientToolResponse (utils.py:78-84), which is precisely the generation trigger we're gating against.
So the withheld outputs sit in local context as a landmine: the next time any code path pushes the full agent context to the session — the any_skipped sync at agent_activity.py:3810 after a later interruption, or a new session getting seeded from agent context after reconnect/handoff — they show up in the diff and get delivered as tool responses at that arbitrary moment, potentially triggering an unprompted spoken reply mid-conversation.
I don't think I'm familiar enough with this codepath to judge the pertinence of this claim though. To be safe, I just reverted realtime changes.
|
|
||
| # commit before any interruption gate: dropping already-executed tool | ||
| # results makes the next inference re-issue the calls (see #3702) | ||
| if _previous_tools_messages: |
There was a problem hiding this comment.
Could we instead commit tool_messages to the agent's chat_ctx right after the tools finish, before creating the reply speech task? Then the commit is guaranteed regardless of what happens to that speech, _previous_tools_messages and the no-reply commit both go away, and the messages keep their natural created_at.
Only the interrupted-during-playout return still needs its own commit.
There was a problem hiding this comment.
Yeah, that makes sense, thanks for pointing it out. I dropped _previous_tools_messages
…time push on auto_tool_reply_generation Address review feedback: - pipeline: commit tool_messages immediately after tools finish instead of threading them through _previous_tools_messages into the next reply task; the commit is now guaranteed regardless of what happens to the reply speech, and the no-reply branch goes away - realtime: only push interrupted tool outputs to the server when the model doesn't auto-generate tool replies; for auto-generating models (Gemini) the push would trigger a spoken reply right after the interruption, so keep the results local-only and warn about possible re-execution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e pipeline path The realtime fix needs a per-model answer for auto_tool_reply_generation (withheld outputs would be delivered by any later full-context sync and could trigger an unprompted generation), so leave that path as it is on main for now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 participants
Fixes #3702
Extends #4877 (credit @CarlosGarciaPro) — covers the remaining interruption path, keeps the realtime server-side context in sync, and adds regression tests.
Problem
Completed tool results only reach the agent's
chat_ctxif the speech carrying them survives its interruption gates. If the user interrupts at the wrong moment, the completedFunctionCall/FunctionCallOutputare silently dropped: the next inference has no record the tool ran, so the LLM re-issues the same call — duplicating side effects for non-idempotent tools (bookings, payments, ...).Two ways this happens:
_previous_tools_messages— is interrupted before reaching the commit site (not covered by fix: preserve completed tool results in chat context after interruption #4877; the path we hit in production).tool_output.outputwere discarded (the variant fix: preserve completed tool results in chat context after interruption #4877 fixes).Changes
All in
agent_activity.py, no API changes:_previous_tools_messagesat the start of the reply turn, before any interruption gate (old site removed;insertorders bycreated_at, so conversation ordering is unchanged).tool_output.output(pipeline and realtime paths, as in fix: preserve completed tool results in chat context after interruption #4877). The realtime path also pushes the outputs to the realtime session, since its server-side state drives the next generation.Tests
Five hermetic tests in
tests/test_tool_results_preserved_on_interruption.py: both pipeline variants (including a tool still in flight at interruption), the realtime path, and the handoff exclusion. The four preservation tests fail onmainand pass with this change. Existing suites andmake checkpass.