IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

[lihaolin88]

Hello, can I ask a question during training, when I run train.py, the system raise an error: IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
The error message is shown below:
'''
Error executing job with overrides: []
Traceback (most recent call last):
File "train.py", line 47, in main
trainer.fit(model, trainset, validset)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 609, in fit
self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
results = self._run_stage()
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
self._run_train()
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
self.fit_loop.run()
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 213, in advance
batch_output = self.batch_loop.run(kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 90, in advance
outputs = self.manual_loop.run(kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/loops/optimization/manual_loop.py", line 110, in advance
training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/pytorch_lightning/strategies/strategy.py", line 378, in training_step
return self.model.training_step(*args, **kwargs)
File "/data/mount_d/MultiPly/code/multiply_model.py", line 192, in training_step
model_outputs = self.model(inputs)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/data/mount_d/MultiPly/code/lib/model/multiply.py", line 455, in forward
weights, transmittance, alphas = render_weight_from_density(t_starts, t_ends, sigmas, ray_indices=ray_indices, n_rays=n_rays)
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/nerfacc/vol_rendering.py", line 391, in render_weight_from_density
packed_info, t_starts, t_ends, sigmas
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/nerfacc/vol_rendering.py", line 633, in forward
packed_info, t_starts, t_ends, sigmas
File "/home/haolin/anaconda3/envs/multiply/lib/python3.7/site-packages/nerfacc/cuda/init.py", line 13, in call_cuda
return getattr(_C, name)(*args, **kwargs)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
'''
I git clone the whole repo and didn't make change, my environment setting is:
cudatoolkit: 11.1
python: 3.7
pytorch: 1.10.0+cu111
torchvision: 0.10.0+cu111
pytorch3d: 0.7.7
kaolin: 0.13.0

Thank you for the reply.

[jzr99]

Hi,

I thought there is some compatibility issue with the environment.

• First of all, your pytorch version is 1.10 which should be companied with torchvision 0.11 according to https://pypi.org/project/torchvision/ (but we recommend you use the same torch (1.12.0) and torchvision (0.13.0) cu113 with us.)
• Second, which nerfacc version do you use? we use 0.5.3. Since there is no prebuild nerfacc for torch 1.10.0 CUDA 111 (even I do not know if nerfacc is compatiable with torch 1.10.0 ) , could you please build it from source following the instruction here Also, there are some prebuild nerfacc to use if you use exactly same torch version (1.12.0+cu113) and torchvision version (0.13.0+cu113) with us.