[Bug]: trtllm-serve crashes with std::bad_cast when eos_token_id is a list (Llama 3.1)

[CristyNel]

System Info

• GPU: NVIDIA GeForce RTX 5090 Blackwell (SM 120)
• CPU: Intel i9-14900K Raptor Lake-S Refresh (32) @ 5.700GHz
• RAM: TeamGroup Delta RGB 64GB DDR5-7600 7600MHz Dual Channel
• OS: Ubuntu 24.04.3 LTS x86_64
• Kernel: 6.14.0-37-generic
• Docker Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2
• Python Version: 3.12.3
• CUDA Version: 12.8

Who can help?

@2ez4bz
@yuanjingx87
@karljang
@greg-kwasniewski1
@Wanli-Jiang
@kaiyux
@nv-guomingz
@nekorobov
@Funatiq
@byshiue

Information

• The official example scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)

Reproduction

# Build the engine for Llama 3.1 (which uses a list for eos_token_id in generation_config.json)

trtllm-build \
  --checkpoint_dir /models/llama3_8b_legacy \
  --output_dir /models/llama3_8b_base_engine \
  --max_batch_size 1 \
  --max_seq_len 3000 \
  --max_num_tokens 6000

#  Start the FastAPI server using the C++ Executor backend

trtllm-serve /models/llama3_8b_base_engine \
  --backend tensorrt \
  --tokenizer /models/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000

#  Send a basic completions request (in a separate terminal)

curl -sS <http://127.0.0.1:8000/v1/completions> \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3_8b_base_engine",
    "prompt": "Say hello.",
    "max_tokens": 10
  }'

Expected behavior

The trtllm-serve wrapper and underlying C++ Executor API should natively parse and accept an array/list of integers for eos_token_id (and pad_token_id) directly from the Hugging Face generation_config.json, correctly mapping them to multiple stop conditions without throwing a C++ exception.

Actual behavior

The server instantly rejects the request and throws a C++ type-casting panic. Because the official Meta config sets "eos_token_id": [128001, 128009], PyBind11 fails to cast the Python list into the single int32_t variable expected by the C++ SamplingConfig.

The server logs the following error to the client:
{"object":"error","message":"Encountered an error in forwardAsync function: std::bad_cast","type":"BadRequestError","param":null,"code":400}

Additional notes

• Impact: This breaks out-of-the-box deployment for the Llama 3/3.1 family when using trtllm-serve in a production environment.
• Constraint: We cannot manually alter generation_config.json to force the list into a single integer.
• Question regarding support: Given that Meta's official Llama 3.1 configurations natively utilize a list for eos_token_id, is this model fully supported in the 1.3.0rc4 Executor API without modifying the upstream JSON?
• Request for Help: If it is supported, what is the officially recommended way to pass or override these stop tokens (e.g., via server startup flags or the API request payload) to safely bypass the C++ list-casting crash while we wait for a permanent patch?

[karljang]

Thanks for the detailed report and reproduction steps.
I attempted to reproduce the crash using RTX 5090 and H200, but unfortunately, I was unsuccessful. :)

My Setup

• GPU: NVIDIA GeForce RTX 5090 Blackwell (SM 120) — same as yours
• Docker Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2
• Python Version: 3.12.3
• CUDA Version: 13.1
• TensorRT-LLM Version: 1.3.0rc2
• Model: Llama-3.1-8B-Instruct (has eos_token_id: [128001, 128008, 128009])

Reproduction Steps

# 1. Convert HF model to TRT-LLM checkpoint
python examples/models/core/llama/convert_checkpoint.py \
    --model_dir ../../models/Llama-3.1-8B-Instruct \
    --output_dir ./checkpoint \
    --dtype bfloat16

# 2. Build engine (matching your exact parameters)
trtllm-build \
    --checkpoint_dir ./checkpoint \
    --output_dir ./engine \
    --max_batch_size 1 \
    --max_seq_len 3000 \
    --max_num_tokens 6000

# 3. Serve from pre-built engine
trtllm-serve ./engine \
    --backend tensorrt \
    --tokenizer ../../models/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 8199 \
    --max_batch_size 1

Test Results

# 4. Send request (same as yours)
curl -sS http://127.0.0.1:8199/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "engine",
        "prompt": "Say hello.",
        "max_tokens": 10
    }'

{"id":"cmpl-4b6c39c5f5324a8a9f186a36ca4b28d5","object":"text_completion","created":1771834172,"model":"engine","choices":[{"index":0,"text":" Welcome to our world of creation.\nIn Seattle,","token_ids":null,"logprobs":null,"context_logits":null,"finish_reason":"length","stop_reason":null,"disaggregated_params":null,"avg_decoded_tokens_per_iter":1.0}],"usage":{"prompt_tokens":4,"total_tokens":14,"completion_tokens":10,"prompt_tokens_details":{"cached_tokens":0}},"prompt_token_ids":null}

# Chat completions
curl -sS http://127.0.0.1:8199/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "engine",
        "messages": [{"role": "user", "content": "Say hello."}],
        "max_tokens": 20
    }'
{"id":"chatcmpl-1213b1359bcb47d2a83e12172de768e6","object":"chat.completion","created":1771834224,"model":"engine","choices":[{"index":0,"message":{"role":"assistant","content":"Hello. Is there something I can help you with?","reasoning_content":"","reasoning":null,"tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null,"mm_embedding_handle":null,"disaggregated_params":null,"avg_decoded_tokens_per_iter":0.9166666865348816}],"usage":{"prompt_tokens":38,"total_tokens":49,"completion_tokens":11,"prompt_tokens_details":{"cached_tokens":0}},"prompt_token_ids":null

Test Resutls with explicit stop_token_ids

I also tested on an H200 with the same steps — same result, no crash.
And please note that the last curl command included stop_token_ids.

# Completions with explicit stop_token_ids
curl -sS http://127.0.0.1:8199/v1/completions \\
    -H "Content-Type: application/json" \
    -d '{
        "model": "engine",
        "prompt": "Say hello.",
        "max_tokens": 20,
        "stop_token_ids": [128001, 128008, 128009]
    }'
{"id":"cmpl-8f0dd5c8034a45f48ecd0fc5b884ced6","object":"text_completion","created":1771834233,"model":"engine","choices":[{"index":0,"text":" Say hello. I already know you. They say hello. They say hello. They already know you","token_ids":null,"logprobs":null,"context_logits":null,"finish_reason":"length","stop_reason":null,"disaggregated_params":null,"avg_decoded_tokens_per_iter":1.0}],"usage":{"prompt_tokens":4,"total_tokens":24,"completion_tokens":20,"prompt_tokens_details":{"cached_tokens":0}},"prompt_token_ids":null}

Test Results with different stop_token_ids

• In this test, I used ‘today’ (token ID 3432) as a stop token, and the stop_reason field displays the token ID precisely.

$ curl -sS http://127.0.0.1:8199/v1/chat/completions     -H "Content-Type: application/json"     -d '{
        "model": "engine",
        "messages": [{"role": "user", "content": "Say hello."}],
        "max_tokens": 200,
        "stop_token_ids": [3432]
    }' | jq

{
  "id": "chatcmpl-ffae00d09e8049048bb67bc77f784eef",
  "object": "chat.completion",
  "created": 1771835143,
  "model": "engine",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello, how can I assist you",
        "reasoning_content": "",
        "reasoning": null,
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": 3432,
      "mm_embedding_handle": null,
      "disaggregated_params": null,
      "avg_decoded_tokens_per_iter": 1.0
    }
  ],
  "usage": {
    "prompt_tokens": 38,
    "total_tokens": 45,
    "completion_tokens": 7,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "prompt_token_ids": null
}

[CristyNel]

# inside TensorRT version 1.3.0rc4 container, with /models and /workspace mounted, and working dir set to /models/Llama-3.1-8B-Instruct port 8000:8000
cd /models/Llama-3.1-8B-Instruct

# 1. Convert HF model to TRT-LLM checkpoint
python /app/tensorrt_llm/examples/models/core/llama/convert_checkpoint.py \
  --model_dir /models/Llama-3.1-8B-Instruct \
  --output_dir ./checkpoint \
  --dtype bfloat16

# 2. Build engine (matching your exact parameters)
trtllm-build \
    --checkpoint_dir ./checkpoint \
    --output_dir ./engine \
    --max_batch_size 1 \
    --max_seq_len 3000 \
    --max_num_tokens 6000


# 3. Serve from pre-built engine
trtllm-serve ./engine \
    --backend tensorrt \
    --tokenizer ./ \
    --host 0.0.0.0 --port 8000 \
    --max_batch_size 1

# output container shell
INFO:     Started server process [1276]
INFO:     Waiting for application startup.
INFO:     Application startup complete.


# 4. Test with curl from host shell
curl -sS http://127.0.0.1:8000/v1/models | jq

# output host shell
  "object": "list",
  "data": [
    {
      "id": "engine",
      "object": "model",
      "created": 1771851634,
      "owned_by": "tensorrt_llm"
    }
  ]
}

# 5. Test with curl completions endpoint
curl -sS --max-time 10 --show-error http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3_8b_base_engine","prompt":"Say hello.","max_tokens":200,"stop_token_ids":[128001,128008,128009]}' | jq

# output host shell
{
  "id": "cmpl-6ccf5261f762421087ff510d13580c74",
  "object": "text_completion",
  "created": 1771853521,
  "model": "llama3_8b_base_engine",
  "choices": [
    {
      "index": 0,
      "text": " Say goodbye. Then try to shake off the lingering flavor that lingers after a conversation with singer-songwriter Dar Williams. Her music transports listeners on intimate journeys. Those fortunate enough to catch her live performance are treated to poignant storytelling, buttery vocals, and whimsical guitar work.\nBorn on April 19, 1967, in Mount Holly, New Jersey, Williams was raised in a bohemian family. She grew up with a strong musical foundation, influenced by artists like Joni Mitchell, Neil Young and Bob Dylan. After taking a few years off from college to travel in Europe and study yoga, Williams developed her unique musical style. Her songs weave tales of everyday people and situations with poetic depth and introspection, creating a sense of empathy and understanding among listeners.\nDar Williams' songwriting has earned her a reputation among both folk and pop audiences. Her introspective and witty songcraft has drawn comparisons to Bronte sisters-esque classic literature. With a career spanning over 30 years, Williams",
      "token_ids": null,
      "logprobs": null,
      "context_logits": null,
      "finish_reason": "length",
      "stop_reason": null,
      "disaggregated_params": null,
      "avg_decoded_tokens_per_iter": 1.0
    }
  ],
  "usage": {
    "prompt_tokens": 4,
    "total_tokens": 204,
    "completion_tokens": 200,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "prompt_token_ids": null
}

Thanks, now i want to try to NVFP4 FP8KV

because disappointed

(pid=118333) 
(pid=118333)        █     █     █▄   ▄█
(pid=118333)  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.15.1.dev0+gf17644344.d20260203
(pid=118333)   █▄█▀ █     █     █     █  model   /models/Llama-3.1-8B-Instruct-NVFP4
(pid=118333)    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(pid=118333) 
(pid=118503)  GPU KV cache size: 296,176 tokens
(pid=118333) INFO:     127.0.0.1:45312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(pid=118333) Avg prompt throughput: 6088.2 tokens/s, Avg generation throughput: 75.4 tokens/s, GPU KV cache usage: 0.0%, Prefix cache hit rate: 66.7%

[karljang]

I'm closing this issue. Please feel free to open a new one if you need~ Thank you!

[CristyNel]

I'm closing this issue. Please feel free to open a new one if you need~ Thank you!

Thank you

[CristyNel]

I'm closing this issue. Please feel free to open a new one if you need~ Thank you!

The issue persist with NVFP4 kv FP8

NNVP4 FP8 KV: std::bad_cast??

# inside TensorRT version 1.3.0rc4 container, with /models and /workspace mounted, and working dir set to /models/Llama-3.1-8B-Instruct port 8000:8000
cd /models/Llama-3.1-8B-Instruct

# Quantize to NVFP4 and export legacy TRT‑LLM checkpoint kv fp8
export DEBUG_VERBOSE=1
export CALIB_SAMPLES=512
export CALIB_BATCH=2
export CALIB_MAX_LEN=8192
mkdir -p ./checkpoint_nvfp4_kvfp8
python /app/tensorrt_llm/examples/quantization/quantize.py \
  --model_dir /models/Llama-3.1-8B-Instruct \
  --qformat nvfp4 \
  --kv_cache_dtype fp8 \
  --dtype bfloat16 \
  --output_dir ./checkpoint_nvfp4_kvfp8 \
  --calib_size 512 \
  --batch_size 2 \
  --calib_max_seq_length 8192  2>&1 | tee ./checkpoint_nvfp4_kvfp8/Lama3.1_8B_TRT-checkpoint_export_nvfp4_fp8kv_debug.log


# Build engine
trtllm-build \
  --checkpoint_dir ./checkpoint_nvfp4_kvfp8 \
  --output_dir ./engine_nvfp4_kvfp8 \
  --max_batch_size 1 \
  --max_seq_len 30000 \
  --max_num_tokens 60000

# Serve from pre-built engine
trtllm-serve ./engine_nvfp4_kvfp8 \
  --backend tensorrt \
  --tokenizer ./ \
  --host 0.0.0.0 --port 8000 \
  --max_batch_size 1



curl -sS http://127.0.0.1:8000/version
# output host shell
{"version":"1.3.0rc4"}

curl -sS http://127.0.0.1:8000/v1/models | jq
# output host shell
{
  "object": "list",
  "data": [
    {
      "id": "engine_nvfp4_kvfp8",
      "object": "model",
      "created": 1771865917,
      "owned_by": "tensorrt_llm"
    }
  ]
}

curl -sS --max-time 10 --show-error http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"engine_nvfp4_kvfp8","prompt":"Say hello.","max_tokens":200,"stop_token_ids":[128001,128008,128009]}' | jq

# output host shell
{
  "object": "error",
  "message": "Encountered an error in forwardAsync function: std::bad_cast",
  "type": "BadRequestError",
  "param": null,
  "code": 400
}

:/models/Llama-3.1-8B-Instruct$ 

ls -h ./checkpoint_nvfp4_kvfp8
Lama3.1_8B_TRT-checkpoint_export_nvfp4_fp8kv_debug.log  config.json  rank0.safetensors

ls -h ./
LICENSE    USE_POLICY.md  checkpoint_nvfp4_kvfp8  engine              generation_config.json            model-00002-of-00004.safetensors  model-00004-of-00004.safetensors  model.safetensors.index.json  special_tokens_map.json  tokenizer_config.json
README.md  checkpoint     config.json             engine_nvfp4_kvfp8  model-00001-of-00004.safetensors  model-00003-of-00004.safetensors  model.cache                       original                      tokenizer.json

ls -h ./engine_nvfp4_kvfp8
config.json  rank0.engine

[CristyNel]

Isn’t it easier to run a modern 40B model than to build an old 8B engine?
IQuest-Coder-V1-40B-Instruct 79.6 GB vs nvidia/Llama-3.1-8B-Instruct-NVFP4 6.05 GB
(pid=238961)
(pid=238961)       █     █     █▄   ▄█
(pid=238961) ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.15.1.dev0+gf17644344.d20260203
(pid=238961)  █▄█▀ █     █     █     █  model   /NVME2T/models/IQuest-Coder-V1-40B-Instruct-NVFP4
(pid=238961)   ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(pid=238961)
(pid=239125)  GPU KV cache size: 19,520 tokens
(pid=239125)  Maximum concurrency for 17,000 tokens per request: 1.15x
(pid=238961) Avg prompt throughput: 1221.3 tokens/s, Avg generation throughput: 23.2 tokens/s, GPU KV cache usage: 63.8%, Prefix cache hit rate: 67.3%
(pid=238961) Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.1 tokens/s, GPU KV cache usage: 65.4%, Prefix cache hit rate: 67.3%
(pid=238961) Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.0 tokens/s, GPU KV cache usage: 66.9%, Prefix cache hit rate: 67.3%

[karljang]

I’ve reopened this issue and will try the NVFP4-related steps you shared.
However, I’m curious to know if you were able to confirm that the non-NVFP4 steps in my comment worked for you. I’m wondering because the NVFP4 might not be the cause of the bad_cast error.

[karljang]

Using the NVFP4 engine, which was generated by the commands you provided, I attempted to reproduce the crash using an RTX 5090 GPU. Unfortunately, I was unsuccessful again. :)

The only change I made was to fix the tokenizer argument in the trtllm-serve command. Here’s the command I used::

trtllm-serve ./engine_nvfp4_kvfp8 \
  --backend tensorrt \
  --tokenizer /models/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max_batch_size 1

[CristyNel]

Using the NVFP4 engine, which was generated by the commands you provided, I attempted to reproduce the crash using an RTX 5090 GPU. Unfortunately, I was unsuccessful again. :)

The only change I made was to fix the tokenizer argument in the trtllm-serve command. Here’s the command I used::

trtllm-serve ./engine_nvfp4_kvfp8
--backend tensorrt
--tokenizer /models/Llama-3.1-8B-Instruct
--host 0.0.0.0 --port 8000
--max_batch_size 1

retry following @karljang's suggestion

# Serve from pre-built engine
trtllm-serve ./engine_nvfp4_kvfp8 \
  --backend tensorrt \
  --tokenizer /models/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max_batch_size 1

# output container shell
INFO:     Started server process [145]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

curl -sS http://127.0.0.1:8000/version
# output host shell
{"version":"1.3.0rc4"}

curl -sS http://127.0.0.1:8000/v1/models | jq
# output host shell
{
  "object": "list",
  "data": [
    {
      "id": "engine_nvfp4_kvfp8",
      "object": "model",
      "created": 1771933581,
      "owned_by": "tensorrt_llm"
    }
  ]
}

curl -sS --max-time 10 --show-error http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"engine_nvfp4_kvfp8","prompt":"Say hello.","max_tokens":200,"stop_token_ids":[128001,128008,128009]}' | jq

# output host shell
{
  "object": "error",
  "message": "Encountered an error in forwardAsync function: std::bad_cast",
  "type": "BadRequestError",
  "param": null,
  "code": 400
}

[CristyNel]

I’ve reopened this issue and will try the NVFP4-related steps you shared. However, I’m curious to know if you were able to confirm that the non-NVFP4 steps in my comment worked for you. I’m wondering because the NVFP4 might not be the cause of the bad_cast error.

It worked after following your base model commands. Using ./ is a cleaner way to organize them, but NVFP4 does not work.

/models/Llama-3.1-8B-Instruct
├── checkpoint
│   ├── config.json
│   └── rank0.safetensors
├── checkpoint_nvfp4_kvfp8
│   ├── config.json
│   ├── Lama3.1_8B_TRT-checkpoint_export_nvfp4_fp8kv_debug.log
│   └── rank0.safetensors
├── config.json
├── engine
│   ├── config.json
│   └── rank0.engine
├── engine_nvfp4_kvfp8
│   ├── config.json
│   └── rank0.engine
├── generation_config.json
├── LICENSE
├── model-00001-of-00004.safetensors
├── model-00002-of-00004.safetensors
├── model-00003-of-00004.safetensors
├── model-00004-of-00004.safetensors
├── model.cache
├── model.safetensors.index.json
├── original
│   ├── consolidated.00.pth
│   ├── params.json
│   └── tokenizer.model
├── README.md
├── special_tokens_map.json
├── tokenizer_config.json
├── tokenizer.json
└── USE_POLICY.md

# 5. Test with curl completions endpoint
curl -sS --max-time 10 --show-error http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3_8b_base_engine","prompt":"Say hello.","max_tokens":200,"stop_token_ids":[128001,128008,128009]}' | jq

# output host shell
{
  "id": "cmpl-6ccf5261f762421087ff510d13580c74",
  "object": "text_completion",
  "created": 1771853521,
  "model": "llama3_8b_base_engine",
  "choices": [
    {
      "index": 0,
      "text": " Say goodbye. Then try to shake off the lingering flavor that lingers after a conversation with singer-songwriter Dar Williams. Her music transports listeners on intimate journeys. Those fortunate enough to catch her live performance are treated to poignant storytelling, buttery vocals, and whimsical guitar work.\nBorn on April 19, 1967, in Mount Holly, New Jersey, Williams was raised in a bohemian family. She grew up with a strong musical foundation, influenced by artists like Joni Mitchell, Neil Young and Bob Dylan. After taking a few years off from college to travel in Europe and study yoga, Williams developed her unique musical style. Her songs weave tales of everyday people and situations with poetic depth and introspection, creating a sense of empathy and understanding among listeners.\nDar Williams' songwriting has earned her a reputation among both folk and pop audiences. Her introspective and witty songcraft has drawn comparisons to Bronte sisters-esque classic literature. With a career spanning over 30 years, Williams",
      "token_ids": null,
      "logprobs": null,
      "context_logits": null,
      "finish_reason": "length",
      "stop_reason": null,
      "disaggregated_params": null,
      "avg_decoded_tokens_per_iter": 1.0
    }
  ],
  "usage": {
    "prompt_tokens": 4,
    "total_tokens": 204,
    "completion_tokens": 200,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "prompt_token_ids": null
}

[karljang]

Thanks for the update! I'm closing this issue again, please feel free to open a new one if you need to.