Pretraining on Val?

[LandDreamer]

Hi,

I noticed that the validation set is used during Sonata pretrain, even though labels are not accessed. I’m concerned this might introduce some data leakage and influence generalization assessment. It might be better to avoid using the validation set at this stage.
https://github.com/Pointcept/Pointcept/blob/5b10bc05b0a3cab8cd5df5ebd142b56bb791ad15/configs/sonata/pretrain-sonata-v1m1-0-base.py#L197

[Gofinge]

Hey, it looks like an old tradition and privilege of self-supervised learning. We also think this is reasonable, as we typically use leak to describe a leak of human annotation. And also, just like language models, our final destination is all data learning; there is no OOD in this scope.

[LandDreamer]

Thank you for your explanation! I understand that using all available data is common in self-supervised learning, and that data leakage is usually associated with label information.

However, I still feel that pretraining only on the training set is more valuable.. Self-supervised methods can hardly cover all possible data, since in real-world applications there will always be new, unseen data—especially when deploying to edge devices. Therefore, what we really want to know is how well a method generalizes to new data.

Thank you again for your response and for your great work on Sonata!