Mixed precision training?

Pytorch 1.6 has native support for automatic mixed precision (AMP) training: https://pytorch.org/blog/pytorch-1.6-released/

Should we take advantage of this? In particular I think the larger batches would be nice for encoder and synthesizer training.

Mozilla TTS tried it and encountered a bug with RNNs to be fixed in a future Pytorch release. mozilla/TTS#486 (comment)

Pytorch 1.7 just released, so it is time to try again. If we add AMP, the implementation must be clean while preserving support for lower versions of PT.

See fatchord/WaveRNN#229 for an example of how to do this.

Closing due to lack of developer interest at this time.
Please comment and reopen if you would like to work on this.

I made a branch that supports mixed precision training. It is not recommended for use at this time.
https://github.com/blue-fish/Real-Time-Voice-Cloning/tree/487_mixed_precision_training

For me, mixed precision training is much slower than without it, and loss is occasionally nan. I also had to set up my Python environment with Anaconda due to a problem with matrix multiplication. pytorch/pytorch#56747 (comment)

Pytorch AMP enabled (Python 3.9.7 with Anaconda, pytorch==1.10.0):

{| Epoch: 1/8 (20/2564) | Loss: nan | 0.24 steps/s | Step: 0k | }
Average execution time over 10 steps:
  Blocking, waiting for batch (threaded) (10/10):  mean:    0ms   std:    0ms
  Data to cuda (10/10):                            mean:    1ms   std:    0ms
  Forward pass (10/10):                            mean:  956ms   std:  151ms
  Loss (10/10):                                    mean:   14ms   std:    3ms
  Backward pass (10/10):                           mean: 3013ms   std:  456ms
  Parameter update (10/10):                        mean:   71ms   std:    5ms
  Extras (visualizations, saving) (10/10):         mean:    0ms   std:    0ms

Same setup without AMP:

{| Epoch: 1/8 (20/2564) | Loss: 5.778 | 0.76 steps/s | Step: 0k | }
Average execution time over 10 steps:
  Blocking, waiting for batch (threaded) (10/10):  mean:    0ms   std:    0ms
  Data to cuda (10/10):                            mean:    1ms   std:    0ms
  Forward pass (10/10):                            mean:  451ms   std:   52ms
  Loss (10/10):                                    mean:   10ms   std:    2ms
  Backward pass (10/10):                           mean:  753ms   std:   73ms
  Parameter update (10/10):                        mean:   31ms   std:    3ms
  Extras (visualizations, saving) (10/10):         mean:    5ms   std:    0ms

Dropping this due to poor performance and lack of interest.