Skip to content

Fix: streaming tool-call arguments dropped when the leading brace ships with the tool name#801

Merged
mikehostetler merged 4 commits into
agentjido:mainfrom
anagrius:fix/streaming-tool-args-leading-brace
Jul 6, 2026
Merged

Fix: streaming tool-call arguments dropped when the leading brace ships with the tool name#801
mikehostetler merged 4 commits into
agentjido:mainfrom
anagrius:fix/streaming-tool-args-leading-brace

Conversation

@anagrius

@anagrius anagrius commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Problem

Some OpenAI-compatible servers (llama.cpp, vLLM, …) begin streaming a tool call's arguments in the same chunk as the function name. This is valid per the OpenAI streaming spec — the first delta looks like:

{"index":0,"id":"","type":"function","function":{"name":"list_metrics","arguments":"{"}}

…and the continuation deltas carry the rest:

{"index":0,"function":{"arguments":"\"database\":\""}}
{"index":0,"function":{"arguments":"dev"}}

{"index":0,"function":{"arguments":"\"}"}}

decode_openai_tool_call_delta/1 matches the first chunk on the "complete tool call" clause (name and arguments present), calls build_openai_tool_call_chunk("list_metrics", "{", …), Jason.decode("{") fails, and the "{" is preserved only as :raw_arguments metadata on the tool-call chunk — it is never added to the per-index argument-fragment buffer.

The continuation fragments then accumulate to "database":"dev","table":"dev"} — valid JSON except the leading brace is missing — so message_tool_call_args/2 fails to decode and falls back to the tool-call chunk's own (empty) arguments. The tool ends up being called with %{}.

OpenAI/Anthropic/etc. are unaffected because their first tool-call chunk sends arguments: "" (empty), so every argument byte arrives via the continuation path.

Fix

Seed the per-index fragment buffer with the tool-call chunk's :raw_arguments (the dropped leading fragment) in ChunkAccumulator.push/2 for :tool_call chunks. The continuation fragments then append to it and the joined JSON parses correctly. Providers that send an empty arguments in the name chunk have an empty raw_arguments, so nothing is seeded and their behavior is unchanged.

Test

Added a regression test in chunk_accumulator_test.exs covering the llama.cpp/vLLM shape (first chunk carries "{", continuation fragments carry the rest). Verified it fails without the fix and passes with it; the full ChunkAccumulator suite is green.

… deltas

Some OpenAI-compatible servers (llama.cpp, vLLM, …) begin streaming a tool
call's `arguments` in the same chunk as the function `name` — valid per the
OpenAI streaming spec, e.g. the first delta is
`{"index":0,"id":...,"function":{"name":"f","arguments":"{"}}`.

`decode_openai_tool_call_delta/1` decodes that first chunk as a complete call,
`Jason.decode("{")` fails, and the `"{"` is kept only as `:raw_arguments` on the
tool-call chunk — it is never added to the per-index fragment buffer. The
continuation fragments then accumulate to `"...":"..."}` with no leading brace,
`message_tool_call_args/2` fails to decode, and the tool runs with `%{}`.

Seed the fragment buffer with the tool-call chunk's `:raw_arguments` so the
continuation fragments append to it and the joined JSON parses. Providers that
send `arguments: ""` in the name chunk (OpenAI, Anthropic, …) are unaffected —
their `raw_arguments` is empty and nothing is seeded.
@anagrius anagrius force-pushed the fix/streaming-tool-args-leading-brace branch from 71552f4 to 9f14873 Compare July 3, 2026 13:33
@anagrius

anagrius commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Without this the lib is broken for many custom providers.

Move the seed_arg_fragment/3 private helper below the final push/2
clause; interleaving it between push/2 clauses tripped the
"clauses with the same name and arity should be grouped together"
warning, which fails CI under --warnings-as-errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikehostetler mikehostetler added the needs_work Changes requested before merge label Jul 6, 2026
@mikehostetler mikehostetler added ready_to_merge and removed needs_work Changes requested before merge labels Jul 6, 2026
@mikehostetler mikehostetler merged commit cc794c7 into agentjido:main Jul 6, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants