Question about LitePT-B / LitePT-L results on nuScenes

[Garfield2005]

I noticed that in both the paper and the released code, the evaluation on nuScenes is only reported for LitePT-S.

From the available results:

• LitePT-S achieves 82.2 mIoU on nuScenes, outperforming PTv3 (80.4) ([Hugging Face])

However, for larger variants:

• LitePT-B (~45M params) and LitePT-L (~85M params)
  there seem to be results reported only on datasets like Structured3D, but not on nuScenes ([Hugging Face])

This leads to my questions:

1. Have you evaluated LitePT-B or LitePT-L on nuScenes?
2. If so, what mIoU do they achieve compared to LitePT-S?
3. Is there a specific reason why only LitePT-S is reported on nuScenes (e.g., efficiency considerations, diminishing returns, or training cost)?

Any clarification would be greatly appreciated.

[ywyue]

Hi @Garfield2005, we use LitePT-S as the main variant for the experiments, as even this smallest version already achieves performance that is better or comparable to PTv3.

For scaling analysis, we report results from LitePT-S -> B -> L on Structured3D, since it is the largest dataset in the evaluation suite and serves as an ideal benchmark for studying scaling behavior.

We also evaluated LitePT-L on additional datasets, including NuScenes. We note that, in accordance with established scaling laws, model scaling should be considered in conjunction with data volume. Increasing the model from LitePT-S to LitePT-L consistently improves performance. As expected, the gain strongly depends on the dataset size, as denoted by the number of points of the dataset (#Points).

On NuScenes (0.98 Billion points), scaling LitePT-S to L improves mIoU by 0.4 while on Structured3D (9.55 Billion points), which is around 10 times larger, the same scaling yields a 1.8 mIoU improvement.

[Garfield2005]

Thank you very much for your prompt response. The explanation is very detailed and helpful.