Understand multi-process programming using 2_augment.py

[huangchieh]

I still confusing the multi-process programming

[huangchieh]

mp.spawn(run, args=(cfg, gen_ab), nprocs=world_size, join=True) would open world_size (4 in my case) independentt processes.

For each process, the data-loader would use num_workers workers to load the data.

dataloader = wds.WebLoader(dataset, num_workers=cfg['num_workers'], batch_size=None, pin_memory=True,
        collate_fn=dl.default_collate, persistent_workers=True)