Fix: streaming tool-call arguments dropped when the leading brace ships with the tool name#801
Merged
mikehostetler merged 4 commits intoJul 6, 2026
Conversation
… deltas
Some OpenAI-compatible servers (llama.cpp, vLLM, …) begin streaming a tool
call's `arguments` in the same chunk as the function `name` — valid per the
OpenAI streaming spec, e.g. the first delta is
`{"index":0,"id":...,"function":{"name":"f","arguments":"{"}}`.
`decode_openai_tool_call_delta/1` decodes that first chunk as a complete call,
`Jason.decode("{")` fails, and the `"{"` is kept only as `:raw_arguments` on the
tool-call chunk — it is never added to the per-index fragment buffer. The
continuation fragments then accumulate to `"...":"..."}` with no leading brace,
`message_tool_call_args/2` fails to decode, and the tool runs with `%{}`.
Seed the fragment buffer with the tool-call chunk's `:raw_arguments` so the
continuation fragments append to it and the joined JSON parses. Providers that
send `arguments: ""` in the name chunk (OpenAI, Anthropic, …) are unaffected —
their `raw_arguments` is empty and nothing is seeded.
71552f4 to
9f14873
Compare
Contributor
Author
|
Without this the lib is broken for many custom providers. |
Move the seed_arg_fragment/3 private helper below the final push/2 clause; interleaving it between push/2 clauses tripped the "clauses with the same name and arity should be grouped together" warning, which fails CI under --warnings-as-errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Some OpenAI-compatible servers (llama.cpp, vLLM, …) begin streaming a tool call's
argumentsin the same chunk as the functionname. This is valid per the OpenAI streaming spec — the first delta looks like:{"index":0,"id":"…","type":"function","function":{"name":"list_metrics","arguments":"{"}}…and the continuation deltas carry the rest:
{"index":0,"function":{"arguments":"\"database\":\""}} {"index":0,"function":{"arguments":"dev"}} … {"index":0,"function":{"arguments":"\"}"}}decode_openai_tool_call_delta/1matches the first chunk on the "complete tool call" clause (name and arguments present), callsbuild_openai_tool_call_chunk("list_metrics", "{", …),Jason.decode("{")fails, and the"{"is preserved only as:raw_argumentsmetadata on the tool-call chunk — it is never added to the per-index argument-fragment buffer.The continuation fragments then accumulate to
"database":"dev","table":"dev"}— valid JSON except the leading brace is missing — somessage_tool_call_args/2fails to decode and falls back to the tool-call chunk's own (empty) arguments. The tool ends up being called with%{}.OpenAI/Anthropic/etc. are unaffected because their first tool-call chunk sends
arguments: ""(empty), so every argument byte arrives via the continuation path.Fix
Seed the per-index fragment buffer with the tool-call chunk's
:raw_arguments(the dropped leading fragment) inChunkAccumulator.push/2for:tool_callchunks. The continuation fragments then append to it and the joined JSON parses correctly. Providers that send an emptyargumentsin the name chunk have an emptyraw_arguments, so nothing is seeded and their behavior is unchanged.Test
Added a regression test in
chunk_accumulator_test.exscovering the llama.cpp/vLLM shape (first chunk carries"{", continuation fragments carry the rest). Verified it fails without the fix and passes with it; the fullChunkAccumulatorsuite is green.