question about peak memory in TinyTL

[Tsingularity]

Hi, thanks for the great work!

We implemented the update-bias-only transfer learning in our own codebase, however, we didn't notice a huge decrease of peak memory / memory usage during fine-tuning as shown in your paper (only <10% decrease vs >90% claimed in ur paper). The command we used to check GPU memory usage is torch.cuda.max_memory_allocated().

We also checked your released codebase, and the only relevant part is this, which is the same as our implementation.

So I am just wondering how could we get this training memory decrease empirically? Or we're using the wrong command to check memory usage?

Thanks!

[han-cai]

Pytorch has its internal memory management system, which usually occupies a larger memory size than it actually needs.
To get the memory decrease on real hardware devices, you need to explicitly manage the memory allocation step. As far as I know, Pytorch currently does not support this feature. So the only way I know to achieve this is to write a customized training engine instead of using Pytorch.

[Tsingularity]

thanks for the clarification!

just curious, is this implemented in your tiny engine? if yes, do u have any plan for releasing the code for it? (i saw on some other issues, people also asked about this)