Invalid JSON produced in streamed tool_call arguments when using Hermes 3 parser

[fonfonya]

Describe the bug
When using the OpenVINO Model Server OpenAI-compatible /v3 endpoint with stream=true and tool_choice="required" (Hermes 3 tool parser), the streamed arguments field sometimes contains JSON fragments that appear to be split or positioned incorrectly. This results in the final concatenated arguments not being valid JSON.

---

To Reproduce

1. Prepare the model repository using export_model.py:

python export_model.py text_generation \
  --source_model Qwen/Qwen3-4B-Instruct-2507 \
  --weight-format int8 \
  --config_file_path models/config.json \
  --model_repository_path models \
  --tool_parser hermes3

2. Launch OVMS:

docker run -d --user $(id -u):$(id -g) --rm \
  -p 8000:8000 \
  -v $(pwd)/models:/models \
  openvino/model_server:weekly \
  --rest_port 8000 \
  --model_repository_path models \
  --source_model Qwen/Qwen3-4B-Instruct-2507 \
  --tool_parser hermes3 \
  --task text_generation \
  --enable_prefix_caching true

3. Client code:

import openai

client = openai.OpenAI(
    api_key="",
    base_url="http://127.0.0.1:8000/v3",
)

output = ""
for chunk in client.chat.completions.create(
    messages=[{"role": "user", "content": "1+1"}],
    model="Qwen/Qwen3-4B-Instruct-2507",
    stream=True,
    temperature=0.0,
    tool_choice="required",
    tools=[
        {
            "type": "function",
            "function": {
                "name": "search",
                "description": "",
                "parameters": {
                    "type": "object",
                    "properties": {"query": {"type": "string"}},
                    "required": ["query"],
                },
            },
        }
    ],
):
    arguments = chunk.choices[0].delta.tool_calls[0].function.arguments
    if arguments is not None:
        output += arguments

print(output)

4. Observed output:

{"query": "1+1"}}\n</tool_call>

---

Expected behavior
The streamed fragments for the arguments field should concatenate into valid JSON.
For example, the expected final output should be:

{"query": "1+1"}

---

Logs

[2025-12-05 07:30:30.243][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"id":"BZS5lI66X","type":"function","index":0,"function":{"name":"search"}}]}}],"created":1764919794,"model":"
Qwen/Qwen3-4B-Instruct-2507","object":"chat.completion.chunk"}
                                                                                                                                                            
[2025-12-05 07:30:30.243][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:30.244][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.252][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:30.355][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:30.358][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.358][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:30.485][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:30.487][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"index":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 07:30:30.487][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the graph
[2025-12-05 07:30:30.487][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.488][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:30.609][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:30.611][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"query"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","o
bject":"chat.completion.chunk"}                                                                                                                             

[2025-12-05 07:30:30.611][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:30.611][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.611][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:30.732][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:30.733][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 07:30:30.733][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:30.733][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.733][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:30.849][149][llm_executor][info][llm_executor.hpp:66] All requests: 1; Scheduled requests: 1; Cache usage 55.6%;
[2025-12-05 07:30:30.849][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:30.855][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" \""}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 07:30:30.855][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:30.855][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:30.855][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.017][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.022][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"1"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","objec
t":"chat.completion.chunk"}


[2025-12-05 07:30:31.022][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:31.022][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:31.022][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.141][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.142][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:31.142][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.261][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.262][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:31.262][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.383][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.385][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"+1\"}}\n"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507"
,"object":"chat.completion.chunk"}


[2025-12-05 07:30:31.385][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:31.385][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:31.385][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.507][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.508][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"</"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-2507","obje
ct":"chat.completion.chunk"}


[2025-12-05 07:30:31.508][152][llm_calculator][debug][http_llm_calculator.cc:143] LLMCalculator  [Node: LLMExecutor] Response prepared, sending it down the
graph
[2025-12-05 07:30:31.508][152][llm_calculator][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end
[2025-12-05 07:30:31.508][152][llm_calculator][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start
[2025-12-05 07:30:31.621][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 07:30:31.622][152][llm_calculator][debug][servable.cpp:226] Generated complete streaming response: data: {"choices":[{"finish_reason":"stop","in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"tool_call>"}}]}}],"created":1764919794,"model":"Qwen/Qwen3-4B-Instruct-250
7","object":"chat.completion.chunk"}

data: [DONE]

---

Configuration

1. OVMS version

   • OpenVINO Model Server: 2025.4.0.15ce0188a
   • OpenVINO backend: 2025.4.0.0rc2

2. OVMS config.json file

3. CPU only

4. Model repository directory structure

5. Model or publicly available similar model that reproduces the issue

---

Additional context

[mzegla]

Looks like cannot detect the end of arguments correctly - the }\n</tool_call> part is correct model output, but it should not be included as arguments. I'll look into that.
Could you rerun it with OVMS launched with --log_level TRACE and paste the same part of logs you already did, but with those additional trace logs. It would show us what the model actually generated, before OVMS started parsing.

[fonfonya]

Here is the TRACE-level log snippet showing the raw model output before OVMS parsing.
For readability, I removed repetitive debug lines such as:

[....][debug][http_llm_calculator.cc:156] LLMCalculator  [Node: LLMExecutor] Process end  
[....][debug][http_llm_calculator.cc:80] LLMCalculator  [Node: LLMExecutor] Process start

Log:

[2025-12-05 16:06:18.674][152][llm_calculator][trace][servable.cpp:173] Pipeline input text: <|im_start|>system
# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"function": {"description": "", "name": "search", "parameters": {"properties": {"query": {"type": "string"}}, "required": ["query"], "type": "object"}}, "t
ype": "function"}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
1+1<|im_end|>
<|im_start|>assistant

[2025-12-05 16:06:18.674][152][llm_calculator][trace][servable.cpp:174] prompt_token_ids: [151644, 8948, 198, 2, 13852, 271, 2610, 1231, 1618, 825, 476, 803
, 5746, 311, 7789, 448, 279, 1196, 3239, 382, 2610, 525, 3897, 448, 729, 32628, 2878, 366, 15918, 1472, 15918, 29, 11874, 9492, 510, 27, 15918, 397, 4913, 1
688, 788, 5212, 4684, 788, 7342, 330, 606, 788, 330, 1836, 497, 330, 13786, 788, 5212, 13193, 788, 5212, 1631, 788, 5212, 1313, 788, 330, 917, 9207, 2137, 3
30, 6279, 788, 4383, 1631, 7914, 330, 1313, 788, 330, 1700, 9207, 2137, 330, 1313, 788, 330, 1688, 16707, 522, 15918, 1339, 2461, 1817, 729, 1618, 11, 470,
264, 2951, 1633, 448, 729, 829, 323, 5977, 2878, 220, 151657, 151658, 11874, 9492, 510, 151657, 198, 4913, 606, 788, 366, 1688, 11494, 8066, 330, 16370, 788
, 366, 2116, 56080, 40432, 31296, 151658, 151645, 198, 151644, 872, 198, 16, 10, 16, 151645, 198, 151644, 77091, 198]

[2025-12-05 16:06:38.819][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [<]
[2025-12-05 16:06:38.934][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [tool_call>
]
[2025-12-05 16:06:39.284][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [{"]
[2025-12-05 16:06:39.407][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [name]
[2025-12-05 16:06:39.530][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [":]
[2025-12-05 16:06:39.658][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [ "]
[2025-12-05 16:06:39.830][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [search]
[2025-12-05 16:06:39.956][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [",]
[2025-12-05 16:06:40.077][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [ "]
[2025-12-05 16:06:40.184][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [arguments]
[2025-12-05 16:06:40.276][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [":]
[2025-12-05 16:06:40.277][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"id":"axJc5IXiE","type":"function","index":0,"function":{"name":"search"}}]}}],"created":1764950778,"model":"
Qwen/Qwen3-4B-Instruct-2507","object":"chat.completion.chunk"}


[2025-12-05 16:06:40.386][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.387][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [ {"]
[2025-12-05 16:06:40.500][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.501][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [query]
[2025-12-05 16:06:40.502][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 16:06:40.612][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.613][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [":]
[2025-12-05 16:06:40.613][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"query"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","o
bject":"chat.completion.chunk"}


[2025-12-05 16:06:40.720][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.721][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [ "]
[2025-12-05 16:06:40.721][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 16:06:40.828][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.834][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [1]
[2025-12-05 16:06:40.834][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":" \""}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","obj
ect":"chat.completion.chunk"}


[2025-12-05 16:06:40.997][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:40.998][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [+1"}}
]
[2025-12-05 16:06:40.998][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"1"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","objec
t":"chat.completion.chunk"}


[2025-12-05 16:06:41.315][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:41.317][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [</]
[2025-12-05 16:06:41.317][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"+1\"}}\n"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507"
,"object":"chat.completion.chunk"}


[2025-12-05 16:06:41.416][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:41.419][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [tool]
[2025-12-05 16:06:41.419][152][llm_calculator][debug][servable.cpp:206] Generated subsequent streaming response: data: {"choices":[{"finish_reason":null,"in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"</"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-2507","obje
ct":"chat.completion.chunk"}


[2025-12-05 16:06:41.517][152][llm_calculator][debug][http_llm_calculator.cc:136] LLMCalculator  [Node: LLMExecutor] Received partial execution results
[2025-12-05 16:06:41.518][149][llm_executor][info][llm_executor.hpp:90] All requests: 0; Scheduled requests: 0;
[2025-12-05 16:06:41.518][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [_call]
[2025-12-05 16:06:41.519][152][llm_calculator][trace][servable.cpp:91] Streamer callback executed with text: [>]
[2025-12-05 16:06:41.519][152][llm_calculator][debug][servable.cpp:226] Generated complete streaming response: data: {"choices":[{"finish_reason":"stop","in
dex":0,"logprobs":null,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"tool_call>"}}]}}],"created":1764950778,"model":"Qwen/Qwen3-4B-Instruct-250
7","object":"chat.completion.chunk"}

data: [DONE]

Please let me know if you'd like the full unfiltered log or a longer excerpt.

[mzegla]

That's okay. I'll reproduce on my end and get back to you early next week.

[mzegla]

The fix is on main now. I'll let you know when our next weekly containing that change is available. Other way is to build OVMS from source.