[Bug]: `assert request.num_output_placeholders >= 0` can fail in async scheduling

[remusao]

Your current environment

Currently using vLLM v0.12.0 (issue also seen on v0.11.2; not sure about older versions) with the official Docker image, on 8xH200 nodes.

🐛 Describe the bug

Using the following command to launch vLLM:

sudo docker run --gpus all --detach \
    -e "VLLM_USE_DEEP_GEMM=1" \
    -e "VLLM_MOE_USE_DEEP_GEMM=0" \
    -e "CUDA_VISIBLE_DEVICES=4,5,6,7" \
    -p 8235:8080 \
    --ipc=host \
    vllm/vllm-openai:v0.12.0 \
    --host=0.0.0.0 \
    --port=8080 \
    --model=Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
    --gpu-memory-utilization=0.95 \
    --load-format=safetensors \
    --max-num-seqs=64 \
    --tensor-parallel-size 4 \
    --seed=0 \
    --served-model-name qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --async_scheduling

It happens that the following assertion fails (I assume only when async scheduling is enabled but I didn't try without; as the assertion can take a long time to fail).

(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845] Traceback (most recent call last):
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 836, in run_engine_core
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 863, in run_busy_loop
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     self._process_engine_step()
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 892, in _process_engine_step
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 443, in step_with_batch_queue
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     engine_core_outputs = self.scheduler.update_from_output(
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 1114, in update_from_output
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     new_token_ids, stopped = self._update_request_with_output(
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 61, in _update_request_with_output
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]     assert request.num_output_placeholders >= 0
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=396) ERROR 12-05 07:26:46 [core.py:845] AssertionError

Unfortunately I do not have a repro for this as the assertion can fail after hours of serving traffic normally so it's pretty uncommon, but from our experience, it will eventually happen (I could see these traces on most nodes). Luckily the node restarts but I thought I'd share since it looks like a potential bug.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Thanks for reporting this!

[njhill]

Thanks for reporting @remusao. This is only based on a hunch but do you know if structured outputs might be in use here? either explicitly in some requests or I'm not sure if they are used implicitly in the tool parsing in this case.

It may also be helpful if you could share a larger chunk of the log around the error/stacktrace.

[njhill]

@remusao another question, are you setting the request_id and/or X-Request-Id header explicitly in any of the requests? There are some known issues related to duplicate ids (particularly with async scheduling) and another hunch is that it might also be related to that. We'll be fixing this very soon via #27987.

[remusao]

Thanks for jumping on this issue @njhill @robertgshaw2-redhat, I will need to double-check but I do not think we rely on vLLM for tool parsing (Edit: we do not rely on vLLM for tool parsing). On the other hand we do use X-Request-Id so that last issue seems relevant (we will try to remove the header and see if that mitigates the crashes; this header is not something we strictly need vLLM to know about).

I can also share some more logs once I’m back at a laptop but I think there was nothing interesting prior to the trace (only the usual logs about throughput) but there was a long trace below about the worker dying.

Edit: Here is a trace with more context before and after the assertion failed.

(APIServer pid=7) INFO:     172.19.0.2:37594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:50536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53052 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53066 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53082 - "GET /health HTTP/1.0" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53116 - "GET /health HTTP/1.0" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53110 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53118 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53124 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:53134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     192.168.0.42:34342 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=7) INFO 11-30 11:50:39 [loggers.py:236] Engine 000: Avg prompt throughput: 868.9 tokens/s, Avg generation throughput: 67.4 tokens/s, Running: 3 reqs, Waiting: 7 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 32.4%
(APIServer pid=7) INFO:     172.19.0.2:36826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     172.19.0.2:36840 - "POST /v1/completions HTTP/1.1" 200 OK
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844] Traceback (most recent call last):
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 835, in run_engine_core
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 862, in run_busy_loop
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     self._process_engine_step()
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 891, in _process_engine_step
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 439, in step_with_batch_queue
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     engine_core_outputs = self.scheduler.update_from_output(
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 1050, in update_from_output
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     new_token_ids, stopped = self._update_request_with_output(
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 55, in _update_request_with_output
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]     assert request.num_output_placeholders >= 0
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=142) ERROR 11-30 11:50:41 [core.py:844] AssertionError
(Worker_TP3 pid=278) INFO 11-30 11:50:41 [multiproc_executor.py:702] Parent process exited, terminating worker
(Worker_TP1 pid=276) INFO 11-30 11:50:41 [multiproc_executor.py:702] Parent process exited, terminating worker
(Worker_TP2 pid=277) INFO 11-30 11:50:41 [multiproc_executor.py:702] Parent process exited, terminating worker
(Worker_TP0 pid=275) INFO 11-30 11:50:41 [multiproc_executor.py:702] Parent process exited, terminating worker
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525] AsyncLLM output_handler failed.
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525] Traceback (most recent call last):
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 477, in output_handler
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525]     outputs = await engine_core.get_output_async()
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 883, in get_output_async
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525]     raise self._format_exception(outputs) from None
(APIServer pid=7) ERROR 11-30 11:50:41 [async_llm.py:525] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] Error in chat completion stream generator.
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] Traceback (most recent call last):
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 611, in chat_completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 351, in completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for prompt_idx, res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 271, in merge_async_iterators
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for item in iterators[0]:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 351, in completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for prompt_idx, res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 271, in merge_async_iterators
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for item in iterators[0]:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 677, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 366, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     return await self.chat_completion_full_generator(
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 1298, in chat_completion_full_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 477, in output_handler
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     outputs = await engine_core.get_output_async()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 883, in get_output_async
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise self._format_exception(outputs) from None
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] Error in chat completion stream generator.
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] Traceback (most recent call last):
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 611, in chat_completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 677, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 366, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     return await self.chat_completion_full_generator(
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 1298, in chat_completion_full_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 611, in chat_completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 351, in completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for prompt_idx, res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 271, in merge_async_iterators
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for item in iterators[0]:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_completion.py", line 351, in completion_stream_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for prompt_idx, res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 271, in merge_async_iterators
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for item in iterators[0]:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 677, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     generator = await handler.create_chat_completion(request, raw_request)
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 366, in create_chat_completion
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     return await self.chat_completion_full_generator(
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 1298, in chat_completion_full_generator
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     async for res in result_generator:
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 423, in generate
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     out = q.get_nowait() or await q.get()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]                             ^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 70, in get
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise output
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 477, in output_handler
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     outputs = await engine_core.get_output_async()
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 883, in get_output_async
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278]     raise self._format_exception(outputs) from None
(APIServer pid=7) ERROR 11-30 11:50:41 [serving_chat.py:1278] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=7) INFO:     172.19.0.2:53094 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:53146 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:36854 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:53124 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:36826 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:36866 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     172.19.0.2:36880 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=7) INFO:     Shutting down
(APIServer pid=7) INFO:     Waiting for application shutdown.
(APIServer pid=7) INFO:     Application shutdown complete.
(APIServer pid=7) INFO:     Finished server process [7]