tf.function retracing warning

[bakhtiyar-k]

Hello,

I tried to use different sets of data (generated by me) for network training, however, for some datasets, I get the following warning, but the training finishes.

WARNING:tensorflow:5 out of the last 5 calls to <function main..train at 0x7f48328aa560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

For other datasets, sometimes the warning shows up and the training crashes right after or after several steps. Sometimes, everything goes ok.

What could be the reason for the warning? All datasets go through the same collection and conversion procedure.