About truncated normal distribution based weight initialization

[developer0hye]

https://github.com/rwightman/pytorch-image-models/blob/e4360e6125bb0bb4279785810c8eb33b40af3ebd/timm/models/vision_transformer.py#L392

https://github.com/rwightman/pytorch-image-models/blob/e4360e6125bb0bb4279785810c8eb33b40af3ebd/timm/models/layers/weight_init.py#L44-L62

Hi, @rwightman

Thanks for sharing your works!

I have a question.

Why do you trunc initialized weight values?

Do you use it for numerical stability?

[rwightman]

@developer0hye moving to discussions, it was to attempt to match initialization for some networks implemented in JAX and Tensorflow (as that is a more common default layer init there)