LLaDA2.0-mini torch error

[AIxyz]

python3 /code/dInfer/benchmarks/benchmark.py --model_name /mnt/tenant-home_speed/shared/models/huggingface/LLaDA2.0-mini--572899f-C8 --gen_len 2048 --block_length 32 --gpu 0 --parallel_decoding threshold --threshold 0.9 --cache prefix --use_bd

I use dinfer v0.2.0 in 1*H800-80G to test LLaDA2.0-mini, but the error as follows, what could be the reason for this?@zheng-da

/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directl
y, please report this to the maintainers of the package that installed pynvml for you.                                                                                                        
  import pynvml  # type: ignore[import]                                                                                                                                                       
INFO 12-21 23:08:34 [__init__.py:216] Automatically detected platform cuda.                                                                                                                   
The input args are listed as follows: Namespace(model_name='/mnt/tenant-home_speed/shared/models/huggingface/LLaDA2.0-mini--572899f-C8', gpu='0', gen_len=2048, prefix_look=0, after_look=0, b
lock_length=32, threshold=0.9, warmup_times=0, low_threshold=0.3, cont_weight=0, parallel_decoding='threshold', use_credit=False, exp_name='exp', cache='prefix', use_tp=False, use_shift=Fals
e, use_bd=True, model_type='llada2')                                                                                                                                                          
started 1 0 0 Namespace(model_name='/mnt/tenant-home_speed/shared/models/huggingface/LLaDA2.0-mini--572899f-C8', gpu='0', gen_len=2048, prefix_look=0, after_look=0, block_length=32, threshol
d=0.9, warmup_times=0, low_threshold=0.3, cont_weight=0, parallel_decoding='threshold', use_credit=False, exp_name='exp', cache='prefix', use_tp=False, use_shift=False, use_bd=True, model_ty
pe='llada2', tp_size=1, port_offset=0)                                                                                                                                                        
WARNING 12-21 23:08:36 [__init__.py:3804] Current vLLM config is not set.                                                                                                                     
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
WARNING 12-21 23:08:36 [__init__.py:3804] Current vLLM config is not set.                                                                                                                     
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0                                                                                                    
INFO 12-21 23:08:36 [parallel_state.py:1165] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0                                                                 
[Loading model]                                                                                                                                                                               
EP Enabled: True                                                                                                                                                                              
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 38.08it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14813/14813 [00:00<00:00, 1478270.36it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:13<00:00,  1.45it/s]
unused_keys []                                                                                                                                                                                
not_inited_keys []                                                                                                                                                                            
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [00:00<00:00, 348.66it/s]
/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:282: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setti
ng `torch.set_float32_matmul_precision('high')` for better performance.                                                                                                                       
  warnings.warn(                                                                                                                                                                              
[rank0]: Traceback (most recent call last):                                                                                                                                                   
[rank0]:   File "/code/dInfer/benchmarks/benchmark.py", line 209, in <module>                                                                                                                 
[rank0]:     main(1, 0, gpus[0], args)                                                                                                                                                        
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context                                                                           
[rank0]:     return func(*args, **kwargs)                                                                                                                                                     
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                     
[rank0]:   File "/code/dInfer/benchmarks/benchmark.py", line 135, in main                                                                                                                     
[rank0]:     dllm.generate(input_ids, gen_length=gen_length, block_length=block_length)                                                                                                       
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank0]:     return func(*args, **kwargs)                                                                                                                                                     
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                     
[rank0]:   File "/code/dInfer/python/dinfer/decoding/generate_uniform.py", line 1074, in naive_batching_generate                                                                              
[rank0]:     self.block_runner.prefill(self.model, x[:, :prefill_length], kv_cache, pos_ids[:, :prefill_length], bd_attn_mask[:,:prefill_length,:prefill_length], self.prefilling_limit, block
_length)                                                                                                                                                                                      
[rank0]:   File "/code/dInfer/python/dinfer/decoding/generate_uniform.py", line 168, in prefill                                                                                               
[rank0]:     output = model(prefilling_x.clone(memory_format=torch.contiguous_format), use_cache=True, attention_mask=attn_mask, position_ids=pos_ids.clone(memory_format=torch.contiguous_for
mat))                                                                                                                                                                                         
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^                                                                                                                                                                                         
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl                                                                        
[rank0]:     return self._call_impl(*args, **kwargs)                                                                                                                                          
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                          
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl                                                                                
[rank0]:     return forward_call(*args, **kwargs)                                                                                                                                             
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                             
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 749, in compile_wrapper                                                                           
[rank0]:     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1                                                                                                            
[rank0]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 923, in _compile_fx_inner
[rank0]:     raise InductorError(e, currentframe()).with_traceback(
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 907, in _compile_fx_inner
[rank0]:     mb_compiled_graph = fx_codegen_and_compile(
[rank0]:                         ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1578, in fx_codegen_and_compile
[rank0]:     return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 1236, in codegen_and_compile
[rank0]:     _recursive_post_grad_passes(gm, is_inference=is_inference)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py", line 504, in _recursive_post_grad_passes
[rank0]:     post_grad_passes(gm, is_inference) 
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 208, in post_grad_passes
[rank0]:     GraphTransformObserver(gm, "decompose_auto_functionalized").apply_graph_pass(
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/fx/passes/graph_transform_observer.py", line 85, in apply_graph_pass
[rank0]:     return pass_fn(self.gm.graph)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1232, in decompose_auto_functionalized
[rank0]:     graph_pass.apply(graph)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/pattern_matcher.py", line 1963, in apply
[rank0]:     entry.apply(m, graph, node)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/pattern_matcher.py", line 1115, in apply
[rank0]:     self.handler(match, *match.args, **match.kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/fx_passes/post_grad.py", line 1230, in _
[rank0]:     match.replace_by_example(decomp, flat_args, run_functional_passes=False)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/pattern_matcher.py", line 309, in replace_by_example
[rank0]:     assert len(graph_with_eager_vals.graph.nodes) == len(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch._inductor.exc.InductorError: AssertionError: 

[rank0]: Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

[rank0]:[W1221 23:09:28.578579446 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see
 https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

[zheng-da]

thanks for reporting the issue. could you provide more information? for example, what is the version of pytorch?

[lundu28]

Hi @AIxyz, currently benchmark.py doesn't support running llada2.0. Please use ‘benchmark_dataset_sglang.py’ for llada2.0 models.

Thanks.

[AIxyz]

Hi, @zheng-da , my torch is 2.8.0+cu128, other info as follows:

I build the docker image from vllm/vllm-openai:v0.10.2 (5791f8642d1b) (22.5GB)
I use "pip install ." from dInfer 932d458

root@57ff4372551c:/vllm-workspace# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0
root@57ff4372551c:/vllm-workspace# 
root@57ff4372551c:/vllm-workspace# python3 -m pip list
Package                           Version       Editable project location
--------------------------------- ------------- -------------------------------------------------
accelerate                        1.10.1
aiofile                           3.9.0
aiofiles                          24.1.0
aiohappyeyeballs                  2.6.1
aiohttp                           3.12.15
aiosignal                         1.4.0
airportsdata                      20250909                                                                                                                                                    
annotated-types                   0.7.0                                                                                                                                                       
anthropic                         0.75.0                                                                                                                                                      
anyio                             4.10.0                                                                                                                                                      
apache-tvm-ffi                    0.1.0b15                                                                                                                                                    
astor                             0.8.1                                                                                                                                                       
asttokens                         3.0.1                                                                                                                                                       
attrs                             25.3.0                                                                                                                                                      
awscrt                            0.27.6                                                                                                                                                      
bitsandbytes                      0.47.0                                                                                                                                                      
blake3                            1.0.5                                                                                                                                                       
blinker                           1.4
blobfile                          3.0.0
boto3                             1.40.30
botocore                          1.40.30
build                             1.3.0
cachetools                        6.2.0
caio                              0.9.24
cbor2                             5.7.0
certifi                           2025.8.3
cffi                              2.0.0
charset-normalizer                3.4.3
click                             8.2.1
cloudpickle                       3.1.1
cmake                             4.1.0
compressed-tensors                0.11.0
cryptography                      3.4.8
cuda-bindings                     13.1.1
cuda-pathfinder                   1.3.3
cuda-python                       13.1.1
cufile-python                     0.1.1
cupy-cuda12x                      13.6.0
datasets                          4.4.1
dbus-python                       1.2.18
decorator                         5.2.1
deep_ep                           1.1.0+e3908bf /vllm-workspace/ep_kernels_workspace/DeepEP
deep_gemm                         2.0.0+ea9c5d9
depyf                             0.19.0
dill                              0.4.0
dinfer                            0.1
diskcache                         5.6.3
distro                            1.9.0
distro-info                       1.1+ubuntu0.2
dnspython                         2.8.0
docstring_parser                  0.17.0
einops                            0.8.1
email-validator                   2.3.0
executing                         2.2.1
fastapi                           0.116.1
fastapi-cli                       0.0.11
fastapi-cloud-cli                 0.1.5
fastrlock                         0.8.3
filelock                          3.19.1
flashinfer-python                 0.4.0
frozendict                        2.4.6
frozenlist                        1.7.0
fsspec                            2025.9.0
gguf                              0.17.1
grpcio                            1.75.1
grpcio-tools                      1.75.1
h11                               0.16.0
hf_transfer                       0.1.9
hf-xet                            1.1.10
httpcore                          1.0.9
httplib2                          0.20.2
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.34.4
humanize                          4.13.0
idna                              3.10
importlib-metadata                4.6.4
infinistore                       0.2.35
interegular                       0.3.3
ipython                           9.8.0
ipython_pygments_lexers           1.1.1
jedi                              0.19.2
jeepney                           0.7.1
Jinja2                            3.1.6
jiter                             0.10.0
jmespath                          1.0.1
jsonschema                        4.25.1
jsonschema-specifications         2025.9.1
keyring                           23.5.0
lark                              1.2.2
launchpadlib                      1.10.16
lazr.restfulclient                0.14.4
lazr.uri                          1.0.6
llguidance                        0.7.30
llvmlite                          0.44.0
lm-format-enforcer                0.11.3
lmcache                           0.3.5
lxml                              6.0.2
markdown-it-py                    4.0.0
MarkupSafe                        3.0.2
matplotlib-inline                 0.2.1
mdurl                             0.1.2
meson                             1.9.0
mistral_common                    1.8.5
modelscope                        1.29.2
more-itertools                    8.10.0
mpmath                            1.3.0
msgpack                           1.1.1
msgspec                           0.19.0
multidict                         6.6.4
multiprocess                      0.70.18
nest-asyncio                      1.6.0
networkx                          3.5
ninja                             1.13.0
numba                             0.61.2
numpy                             2.2.6
nvidia-cublas-cu12                12.8.4.1
nvidia-cuda-cupti-cu12            12.8.90
nvidia-cuda-nvrtc-cu12            12.8.93
nvidia-cuda-runtime-cu12          12.8.90
nvidia-cudnn-cu12                 9.10.2.21
nvidia-cudnn-frontend             1.14.1
nvidia-cufft-cu12                 11.3.3.83
nvidia-cufile-cu12                1.13.1.3
nvidia-curand-cu12                10.3.9.90
nvidia-cusolver-cu12              11.7.3.90
nvidia-cusparse-cu12              12.5.8.93
nvidia-cusparselt-cu12            0.7.1
nvidia-cutlass-dsl                4.2.1
nvidia-ml-py                      13.580.82
nvidia-nccl-cu12                  2.27.3
nvidia-nvjitlink-cu12             12.8.93
nvidia-nvtx-cu12                  12.8.90
nvtx                              0.2.13
oauthlib                          3.2.0
openai                            1.99.1
openai-harmony                    0.0.4
opencv-python-headless            4.12.0.88
orjson                            3.11.5
outlines                          0.1.11
outlines_core                     0.2.11
packaging                         25.0
pandas                            2.3.3
parso                             0.8.5
partial-json-parser               0.2.1.1.post6
pexpect                           4.9.0
pillow                            11.3.0
pip                               25.2
pplx-kernels                      0.0.1         /vllm-workspace/ep_kernels_workspace/pplx-kernels
prometheus_client                 0.22.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit                    3.0.52
propcache                         0.3.2
protobuf                          6.32.1
psutil                            7.0.0
ptyprocess                        0.7.0
pure_eval                         0.2.3
py-cpuinfo                        9.0.0
py-spy                            0.4.1
pyarrow                           22.0.0
pybase64                          1.4.2
pybind11                          3.0.1
pycountry                         24.6.1
pycparser                         2.23
pycryptodomex                     3.23.0
pydantic                          2.11.9
pydantic_core                     2.33.2
pydantic-extra-types              2.10.5
Pygments                          2.19.2
PyGObject                         3.42.1
PyJWT                             2.3.0
pynvml                            13.0.1
pyparsing                         2.4.7
pyproject_hooks                   1.2.0
python-apt                        2.4.0+ubuntu4
python-dateutil                   2.9.0.post0
python-dotenv                     1.1.1
python-json-logger                3.3.0
python-multipart                  0.0.20
pytz                              2025.2
PyYAML                            6.0.2
pyzmq                             27.1.0
ray                               2.49.1
redis                             6.4.0
referencing                       0.36.2
regex                             2025.9.1
requests                          2.32.5
rich                              14.1.0
rich-toolkit                      0.15.1
rignore                           0.6.4
rpds-py                           0.27.1
runai-model-streamer              0.14.0
runai-model-streamer-s3           0.14.0
s3transfer                        0.14.0
safetensors                       0.6.2
scipy                             1.16.2
SecretStorage                     3.3.1
sentencepiece                     0.2.1
sentry-sdk                        2.37.1
setproctitle                      1.3.7
setuptools                        79.0.1
setuptools-scm                    9.2.0
sgl-kernel                        0.3.15
sglang                            0.5.3.post1
shellingham                       1.5.4
six                               1.17.0
sniffio                           1.3.1
sortedcontainers                  2.4.0
soundfile                         0.13.1
soxr                              1.0.0
ssh-import-id                     5.11
stack-data                        0.6.3
starlette                         0.47.3
sympy                             1.14.0
tabulate                          0.9.0
tiktoken                          0.11.0
timm                              1.0.16
tokenizers                        0.22.0
torch                             2.8.0+cu128
torch_memory_saver                0.0.9rc2
torchao                           0.9.0
torchaudio                        2.8.0+cu128
torchvision                       0.23.0+cu128
tqdm                              4.67.1
traitlets                         5.14.3
transformers                      4.57.0
triton                            3.4.0
typer                             0.17.4
typing_extensions                 4.15.0
typing-inspection                 0.4.1
tzdata                            2025.3
unattended-upgrades               0.1
urllib3                           2.5.0
uv                                0.8.17
uvicorn                           0.35.0
uvloop                            0.21.0
vllm                              0.10.2
wadllib                           1.3.6
watchfiles                        1.1.0
wcwidth                           0.2.14
websockets                        15.0.1
wheel                             0.45.1
xformers                          0.0.32.post1
xgrammar                          0.1.23
xxhash                            3.6.0
yarl                              1.20.1
zipp                              1.0.0
root@57ff4372551c:/vllm-workspace# 

[AIxyz]

Hi @lundu28 , when I use benchmark_dataset_sglang.py, the error info as follows

date && python3 /code/dInfer/benchmarks/benchmark_dataset_sglang.py --model_name /mnt/tenant-home_speed/shared/models/huggingface/LLaDA2.0-mini--572899f-C8 --gen_len 2048 --block_length 32 --gpu 0 --parallel_decoding threshold --threshold 0.9 --cache prefix --use_bd --dataset /model/LFS/dataset.json && date

[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/dinfer/decoding/diffusion_runner.py", line 298, in __init__
[rank0]:     self.capture()
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/dinfer/decoding/diffusion_runner.py", line 380, in capture
[rank0]:     ) = self.capture_one_batch_size(bs, forward, True, length, cache_length, use_mask = True)
[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/dinfer/decoding/diffusion_runner.py", line 428, in capture_one_batch_size
[rank0]:     input_ids = self.input_ids[:num_tokens].view(bs, num_tokens_per_bs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: shape '[1, 64]' is invalid for input of size 32

My /model/LFS/dataset.json as follows:

64 * this_item

{"prompt": "Lily can run 12 kilometers per hour for 4 hours. After that, she can run 6 kilometers per hour. How many kilometers can she run in 8 hours?"}

[AIxyz]

@lundu28 I find the reason, the prompt is too short in dataset.json.

When I use the data.json as follows, it is successful, thanks.

[AIxyz]

When I use 8 GPUs, I got extra traceback of in /usr/local/lib/python3.12/dist-packages/sglang/srt/distributed/device_communicators/custom_all_reduce.py

date && python3 /code/dInfer/benchmarks/benchmark_dataset_sglang.py --model_name /mnt/tenant-home_speed/shared/models/huggingface/LLaDA2.0-mini--572899f-C8 --gen_len 2048 --block_length 32 --gpu 0,1,2,3,4,5,6,7 --parallel_decoding threshold --threshold 0.9 --cache prefix --use_bd --dataset $PWD/data.json && date