loss diverged to nan when training vits with fp16 option enabled.

[ariacat3366]

I trained VITS using the original dataset with the fp16 option enabled (--use_amp true), but after turning the training around for a few days, the loss values diverged to NAN in the process.
I think the exception handling when values outside the range of fp16 is not working properly.
I did not see this error when the fp16 option was not enabled for the same data set.

[kan-bayashi]

Thank you for your report. Actually, I have not tried fp16 setting so much.
Do you have any comments about doubtful points?

[ariacat3366]

To be honest, the learning process was going well until the middle of the program, so I am not sure what exactly is wrong.
jaywalnut310/vits#15
jaywalnut310/vits#48
These are the official implementations, and a bug was reported here as well, where learning was not possible or loss was NaN when fp16 was enabled.

In my case, I modified espnet's VITS implementation slightly based on the PR, to be able to learn... but I don't know how it effect to result.
jaywalnut310/vits#34