error converting StreamPETR to onnx

[rod409]

Follow the steps to convert StreamPETR to onnx, I get two error. The first I can get past.

The errors occur when converting the head of the model

CUBLAS_WORKSPACE_CONFIG=:4096:8 python tools/pth2onnx.py projects/configs/test_speed/stream_petr_r50_704_bs2_seq_428q_nui_speed_test.py --section pts_head_memory

Traceback (most recent call last):
File "tools/pth2onnx.py", line 302, in
main()
File "tools/pth2onnx.py", line 284, in main
torch.onnx.export(
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/init.py", line 275, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
_model_to_graph(model, args, verbose, input_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 463, in _model_to_graph
graph = _optimize_graph(graph, operator_export_type,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 200, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/init.py", line 313, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 994, in _run_symbolic_function
return symbolic_fn(g, *inputs, **attrs)
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py", line 1859, in slice
raise RuntimeError("step!=1 is currently not supported")
RuntimeError: step!=1 is currently not supported

I got past this first one by setting the opset to 11 in the pth2onnx.py file. After doing so I run into this error

Traceback (most recent call last):
File "tools/pth2onnx.py", line 303, in
main()
File "tools/pth2onnx.py", line 284, in main
torch.onnx.export(
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/init.py", line 275, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
_model_to_graph(model, args, verbose, input_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 501, in _model_to_graph
params_dict = torch._C._jit_pass_onnx_constant_fold(graph, params_dict,
RuntimeError: expected scalar type Long but found Float

Are there additional steps or requirements in the setup? I'm not sure what is causing a type issue here.

[jin-yc10]

Hi @rod409, could you please provide the versions of the packages you are using? Like torch, onnx, onnxruntime etc

[rod409]

Hi @rod409, could you please provide the versions of the packages you are using? Like torch, onnx, onnxruntime etc

Here are package versions I have

python 3.8.20
torch: 1.9.0+cu111
torchvision: 0.10.0+cu111
torchaudio: 0.9.0+cu111
onnx: 1.17.0
onnxruntime: 1.19.2
onnxsim: 0.4.36
mmdet3d: 1.0.0rc6
mmdet: 2.28.2
mmcv-full: 1.6.0
mmsegmentation: 0.30.0
flash-attn: 0.2.2
numpy: 1.23.5
ipython: 8.12.3
fvcore: 0.1.5.post20221221

[rod409]

I also tried to upgrade to pytorch 1.10. I got different errors related to dimensions.

torch 1.10 output

[W shape_type_inference.cpp:434] Warning: Constant folding in symbolic shape inference fails: Dimension out of range (expected to be in range of [-2, 1], but got 2)
Exception raised from maybe_wrap_dim at ../c10/core/WrapDimMinimal.h:33 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7799f1a39d62 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: + 0x1667052 (0x779a44467052 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #2: at::native::unsqueeze(at::Tensor const&, long) + 0x3b (0x779a4447803b in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #3: + 0x1e3091f (0x779a44c3091f in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #4: at::_ops::unsqueeze::redispatch(c10::DispatchKeySet, at::Tensor const&, long) + 0xad (0x779a44a6315d in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #5: + 0x38492d6 (0x779a466492d6 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: + 0x3849472 (0x779a46649472 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: at::_ops::unsqueeze::redispatch(c10::DispatchKeySet, at::Tensor const&, long) + 0xad (0x779a44a6315d in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #8: + 0x34c67ec (0x779a462c67ec in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #9: + 0x34c6cb2 (0x779a462c6cb2 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::_ops::unsqueeze::call(at::Tensor const&, long) + 0x12d (0x779a44a8dacd in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #11: torch::jit::onnx_constant_fold::runTorchBackendForOnnx(torch::jit::Node const*, std::vector<at::Tensor, std::allocatorat::Tensor >&, int) + 0x85c (0x779aeef74f1c in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #12: + 0xbbd732 (0x779aeefbd732 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #13: torch::jit::ONNXShapeTypeInference(torch::jit::Node*, std::map<std::string, c10::IValue, std::lessstd::string, std::allocator<std::pair<std::string const, c10::IValue> > > const&, int) + 0xa8e (0x779aeefc2e8e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #14: + 0xb34a09 (0x779aeef34a09 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #15: + 0x2a534e (0x779aee6a534e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)

frame #20: python() [0x4f563b]
frame #38: + 0xb6c2bb (0x779aeef6c2bb in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #39: torch::jit::NodeToONNX(torch::jit::Node*, torch::jit::Block*, torch::onnx::OperatorExportTypes, std::unordered_map<torch::jit::Value*, torch::jit::Value*, std::hashtorch::jit::Value*, std::equal_totorch::jit::Value*, std::allocator<std::pair<torch::jit::Value* const, torch::jit::Value*> > >&) + 0x515 (0x779aeef66bd5 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #40: torch::jit::BlockToONNX(torch::jit::Block*, torch::jit::Block*, torch::onnx::OperatorExportTypes, std::unordered_map<torch::jit::Value*, torch::jit::Value*, std::hashtorch::jit::Value*, std::equal_totorch::jit::Value*, std::allocator<std::pair<torch::jit::Value* const, torch::jit::Value*> > >&, bool) + 0x146 (0x779aeef67f96 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #41: torch::jit::ToONNX(std::shared_ptrtorch::jit::Graph&, torch::onnx::OperatorExportTypes) + 0x50e (0x779aeef6886e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #42: + 0xb5d893 (0x779aeef5d893 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #43: + 0x2a534e (0x779aee6a534e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
(function ComputeConstantFolding)
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
[W shape_type_inference.cpp:434] Warning: Constant folding in symbolic shape inference fails: Dimension out of range (expected to be in range of [-2, 1], but got 2)
Exception raised from maybe_wrap_dim at ../c10/core/WrapDimMinimal.h:33 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7799f1a39d62 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: + 0x1667052 (0x779a44467052 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #2: at::native::unsqueeze(at::Tensor const&, long) + 0x3b (0x779a4447803b in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #3: + 0x1e3091f (0x779a44c3091f in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #4: at::_ops::unsqueeze::redispatch(c10::DispatchKeySet, at::Tensor const&, long) + 0xad (0x779a44a6315d in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #5: + 0x38492d6 (0x779a466492d6 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: + 0x3849472 (0x779a46649472 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: at::_ops::unsqueeze::redispatch(c10::DispatchKeySet, at::Tensor const&, long) + 0xad (0x779a44a6315d in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #8: + 0x34c67ec (0x779a462c67ec in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #9: + 0x34c6cb2 (0x779a462c6cb2 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::_ops::unsqueeze::call(at::Tensor const&, long) + 0x12d (0x779a44a8dacd in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #11: torch::jit::onnx_constant_fold::runTorchBackendForOnnx(torch::jit::Node const*, std::vector<at::Tensor, std::allocatorat::Tensor >&, int) + 0x85c (0x779aeef74f1c in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #12: + 0xbbd732 (0x779aeefbd732 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #13: torch::jit::ONNXShapeTypeInference(torch::jit::Node*, std::map<std::string, c10::IValue, std::lessstd::string, std::allocator<std::pair<std::string const, c10::IValue> > > const&, int) + 0xa8e (0x779aeefc2e8e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #14: + 0xbc4994 (0x779aeefc4994 in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #15: + 0xb3519c (0x779aeef3519c in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #16: + 0x2a534e (0x779aee6a534e in /home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)

frame #40: python() [0x5a5da1]
frame #41: python() [0x5a4daf]
frame #42: python() [0x45c624]
frame #44: python() [0x44fcae]
frame #46: + 0x29d90 (0x779af2829d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #47: __libc_start_main + 0x80 (0x779af2829e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #48: python() [0x579f8d]
(function ComputeConstantFolding)
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
projects.mmdet3d_plugin
Traceback (most recent call last):
File "tools/pth2onnx.py", line 303, in
main()
File "tools/pth2onnx.py", line 284, in main
torch.onnx.export(
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/init.py", line 316, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 107, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 724, in _export
_model_to_graph(model, args, verbose, input_names,
File "/home/rod/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/onnx/utils.py", line 544, in _model_to_graph
params_dict = torch._C._jit_pass_onnx_constant_fold(graph, params_dict,
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

Here are the packages in this environment:

python 3.8.20
torch: 1.10.1+cu111
torchvision: 0.11.2+cu111
torchaudio: 0.10.1+cu111
onnx: 1.17.0
onnxruntime: 1.19.2
mmdet3d: 1.0.0rc6
mmdet: 2.28.2
mmcv-full: 1.6.0
mmsegmentation: 0.30.0
flash-attn: 0.2.5
numpy: 1.23.5
ipython: 8.12.3
fvcore:: 0.1.5.post20221221

[jin-yc10]

Can you try torch 1.12?

[rod409]

Can you try torch 1.12?

Thanks I got it to work with 1.12.

[ZgyIsNotAvailable]

Hello, @rod409! I have met this problem too. Could you please show me the versions of the packages you are using in Torch 1.12? Thanks!

[rod409]

Hello, @rod409! I have met this problem too. Could you please show me the versions of the packages you are using in Torch 1.12? Thanks!

it is almost the same as before just with updated torch packages.

python 3.8.20
torch: 1.12.1+cu116
torchvision: 0.13.1+cu116
torchaudio: 0.12.1+cu116
onnx: 1.17.0
onnxruntime: 1.19.2
mmdet3d: 1.0.0rc6
mmdet: 2.28.2
mmcv-full: 1.6.0
mmsegmentation: 0.30.0
flash-attn: 0.2.5
numpy: 1.23.5
ipython: 8.12.3
fvcore:: 0.1.5.post20221221

[ZgyIsNotAvailable]

Hello, @rod409! I have met this problem too. Could you please show me the versions of the packages you are using in Torch 1.12? Thanks!

it is almost the same as before just with updated torch packages.

python 3.8.20
torch: 1.12.1+cu116
torchvision: 0.13.1+cu116
torchaudio: 0.12.1+cu116
onnx: 1.17.0
onnxruntime: 1.19.2
mmdet3d: 1.0.0rc6
mmdet: 2.28.2
mmcv-full: 1.6.0
mmsegmentation: 0.30.0
flash-attn: 0.2.5
numpy: 1.23.5
ipython: 8.12.3
fvcore:: 0.1.5.post20221221

I'm very surprised that the python packages in my conda env are exactly the same as those you list, but I still can't run it.

[ZgyIsNotAvailable]

I have created a new conda env and config it according to StreamPETR/docs/setup.md with torch 1.9，and then updated to torch1.12.1+cu116. I have succeeded after reinstall numpy by conda install. I think the problem may be that the version of mmcv-full、numpy and numba match. @rod409 `s version reference is very useful. Thank you so much, @rod409 !