LMDB getting slower after iterations

[mikezhang95]

I customize a dataset with LMDB format. Then I train the model in pytorch's DDP mode with 4 GPUs.

The training process is suddenly going down after some iterations (about 6 times slower). The memory doesn't change a lot.

Do you know the possible reasons?

[bring728]

i have same problem..