[Bug]: [NCC_INLA001] neuronx-cc fails to compile Llama-3.3-70B context_encoding_model with NxDI 0.7 — "type must be boolean, but is null"

[EzioEzi0]

Your current environment

bug Description:

neuronx-cc fails to compile context_encoding_model HLO for Llama-3.3-70B-Instruct when using vllm-neuron 0.3.0 + NxDI 0.7. The token_generation_model compiles successfully, but multiple context_encoding_model buckets fail with an internal error.

Error:

[INTERNAL_ERROR] [NCC_INLA001] Unhandled exception with message:
[json.exception.type_error.302] type must be boolean, but is null
What I've tried (all fail with the same error):

TP=32, max_model_len=8192
TP=32, max_model_len=4096
TP=16, max_model_len=4096
The bug is in the context_encoding_model HLO compilation path — it's not sensitive to TP size or sequence length. The token_generation_model always compiles fine.

Reproduction:

from vllm import LLM, SamplingParams

llm = LLM(
model="meta-llama/Llama-3.3-70B-Instruct",
max_num_seqs=4,
max_model_len=4096,
block_size=32,
num_gpu_blocks_override=1024,
tensor_parallel_size=32,
)
outputs = llm.generate(["Hello, my name is"], SamplingParams(temperature=0.8))

export NEURON_RT_VISIBLE_CORES=0-31
export VLLM_PLUGINS=neuron
python test.py
Failed compiler invocation (from logs):

neuronx-cc compile --framework=XLA
/tmp/nxd_model/context_encoding_model/tp0_bk18/model.MODULE*.hlo_module.pb
--target=trn1 --auto-cast=none --model-type=transformer
--tensorizer-options=--enable-ccop-compute-overlap --cc-pipeline-tiling-factor=2 --vectorize-strided-dma
--lnc=1 -O1
--internal-hlo2tensorizer-options= --modular-flow-mac-threshold=10 --verify-hlo=true
Failed buckets include: _tp0_bk16, _tp0_bk17, _tp0_bk18, _tp0_bk19, _tp0_bk25–_tp0_bk29 depending on configuration.

The output of python collect_env.py

Python: 3.12.3
PyTorch: 2.9.0+cu128
OS: Linux-6.17.0-1007-aws-x86_64-with-glibc2.39

Instance Type

trn1.32xlarge

Python Environment (pip list | grep -E "torch|neuron|nki|vllm|nxdi|nixl")

libneuronxla 2.2.14584.0+06ac23d1
neuronx-cc 2.22.12471.0+b4a00d10
neuronx-distributed 0.16.25997+f431c02e
neuronx-distributed-inference 0.7.15063+bafa28d5
optimum-neuron 0.4.3
torch 2.9.0
torch-neuronx 2.9.0.2.11.19912+e48cd891
torch-xla 2.9.0
torchaudio 2.9.0
torchvision 0.24.0
vllm 0.13.0
vllm-neuron 0.3.0

🐛 Describe the bug

neuronx-cc fails to compile context_encoding_model HLO for Llama-3.3-70B-Instruct using vllm-neuron 0.3.0 + NxDI 0.7 on trn1.32xlarge. The token_generation_model compiles fine — all buckets pass. But context_encoding_model consistently fails on several buckets with:

[INTERNAL_ERROR] [NCC_INLA001] Unhandled exception with message: 
[json.exception.type_error.302] type must be boolean, but is null

Tried TP=32/16, max_model_len=8192/4096 — same error every time. The bug is in the context_encoding HLO compilation path, not sensitive to TP size or sequence length.

# Minimal reproduction
from vllm import LLM, SamplingParams

# Run with: NEURON_RT_VISIBLE_CORES=0-31 VLLM_PLUGINS=neuron python test.py
llm = LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    max_num_seqs=4,
    max_model_len=4096,
    block_size=32,
    num_gpu_blocks_override=1024,
    tensor_parallel_size=32,
)
outputs = llm.generate(["Hello, my name is"], SamplingParams(temperature=0.8))

Failed compiler invocation from logs:

neuronx-cc compile --framework=XLA \
  /tmp/nxd_model/context_encoding_model/_tp0_bk18/model.MODULE_*.hlo_module.pb \
  --target=trn1 --auto-cast=none --model-type=transformer \
  --tensorizer-options=--enable-ccop-compute-overlap --cc-pipeline-tiling-factor=2 --vectorize-strided-dma \
  --lnc=1 -O1 \
  --internal-hlo2tensorizer-options= --modular-flow-mac-threshold=10 --verify-hlo=true

Failed buckets: _tp0_bk16–_tp0_bk19, _tp0_bk25–_tp0_bk29 depending on config.

Python: 3.12.3
PyTorch: 2.9.0+cu128
OS: Linux-6.17.0-1007-aws-x86_64-with-glibc2.39

trn1.32xlarge

libneuronxla 2.2.14584.0+06ac23d1
neuronx-cc 2.22.12471.0+b4a00d10
neuronx-distributed 0.16.25997+f431c02e
neuronx-distributed-inference 0.7.15063+bafa28d5
optimum-neuron 0.4.3
torch 2.9.0
torch-neuronx 2.9.0.2.11.19912+e48cd891
torch-xla 2.9.0
torchaudio 2.9.0
torchvision 0.24.0
vllm 0.13.0
vllm-neuron 0.3.0

Before submitting a new issue...

• Make sure you already searched for relevant issues.

[srivats03]

Hi @EzioEzi0, Thank you for filing the issue! We will take a look and get back to you.

[feiwx-cloud]

Hi @EzioEzi0, we've been able to reproduce and investigate this with the latest Neuron SDK release (2.28.0). Here's what we found:

• ✅ The issue does not reproduce on trn2. It runs successfully there.
• ❌ The failure does reproduce on trn1, and we have filed a ticket with our compiler team for further investigation.

In the meantime, would you be able to run your workload on trn2 with the Neuron SDK 2.28.0 as a temporary workaround?
We'll post updates here as the investigation progresses.