[BUG] Failed to start training with efficientformer_v2

[hacktmz]

efficientformerv2_s2 uses an image size of 224. When I don’t use pre-training parameters and change the input image size to 480 or others, an error will be reported

python3 -u ./train.py /gpu1-data/datasets --num-classes 58 --model efficientformerv2_s2 -b 48 --img-size 480 --sched step --epochs 300 --lr .001 -j 8 --opt adamp --weight-decay 1e-5 --decay-epochs 2 --decay-rate .91 --warmup-lr 1e-6 --warmup-epochs 1 --remode pixel --reprob 0.15 --amp
Training with a single process on 1 device (cuda:0).
/opt/conda/lib/python3.7/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Model efficientformerv2_s2 created, param count:12165636
Data processing configuration for current model + dataset:
input_size: (3, 480, 480)
interpolation: bicubic
mean: (0.485, 0.456, 0.406)
std: (0.229, 0.224, 0.225)
crop_pct: 0.95
crop_mode: center
Using native Torch AMP. Training in mixed precision.
Scheduled epochs: 300. LR stepped per epoch.
Traceback (most recent call last):
File "./train.py", line 1037, in
main()
File "./train.py", line 774, in main
mixup_fn=mixup_fn,
File "./train.py", line 876, in train_one_epoch
output = model(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/gpu2-data/zhenty/pytorch-image-models/timm/models/efficientformer_v2.py", line 645, in forward
x = self.forward_features(x)
File "/gpu2-data/zhenty/pytorch-image-models/timm/models/efficientformer_v2.py", line 627, in forward_features
x = self.stages(x)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/gpu2-data/zhenty/pytorch-image-models/timm/models/efficientformer_v2.py", line 505, in forward
x = self.blocks(x)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/gpu2-data/zhenty/pytorch-image-models/timm/models/efficientformer_v2.py", line 422, in forward
x = x + self.drop_path1(self.ls1(self.token_mixer(x)))
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/gpu2-data/zhenty/pytorch-image-models/timm/models/efficientformer_v2.py", line 160, in forward
q = self.q(x).reshape(B, self.num_heads, -1, self.N).permute(0, 1, 3, 2)
RuntimeError: shape '[48, 8, -1, 49]' is invalid for input of size 2764800

[rwightman]

@hacktmz position embeddings are fixed to a given resolution, you can't change the input size of these models (and many other transformer based arch to to this sort of issue).

It might be possible to write some additional code to resize the posembed so can be adapted to other resolutions, sometimes a drop in performance without fine-tuning though, contributions for that welcome.