[DEGRADATION] I see significant performance degradation with all TIMM models compare to TorchVision on iMet dataset

[Borda]

Describe the bug

Not sure if the problem is my (user) side but running the same pipeline with the same dataset I getting very different F1 performance measures, see the following notebook with the only switch between TV and TIMM model
https://github.com/Borda/kaggle_iMet-collection/blob/main/notebooks/iMet-with-Lightning.ipynb

To Reproduce
Steps to reproduce the behavior:

• used dataset - https://www.kaggle.com/c/imet-2021-fgvc8
• with timm.create_model('resnet50', pretrained=True, num_classes=dm.num_classes) I get f1 about 24%
• with net = LitResnet(arch='resnet50', num_classes=dm.num_classes) I get f1 about 42% ; the model is implemented as
  https://github.com/Borda/kaggle_iMet-collection/blob/d1757e5bcc50065b0f92bddb6e3b40924f2c8819/kaggle_imet/models.py#L11-L22

Expected behavior
there performances shall be almost the same, not difference as double

Desktop (please complete the following information):

• OS: google Colab
• This repository version: 0.4.9

Additional context
Add any other context about the problem here.