About the fid of imagenet64_cond

[encoreus]

excuse me, I trained a tarflow model of imagenet64_cond, but I can only get a FID of 6.27, my evaluate_fid.py setting is:

torchrun --standalone --nproc_per_node=4 evaluate_fid.py
--data=/tar_flow/data
--logdir=runs_original_flow/imagenet64_cond/
--ckpt_file=cond_imagenet_64/imagenet_model_2_768_8_8_0.05.pth
--dataset=imagenet
--img_size=64
--channel_size=3
--patch_size=2
--channels=768
--blocks=8
--layers_per_block=8
--noise_std=0.05
--nvp
--cfg=2.3
--attn_temp=1.0
--batch_size=512
--num_samples=50000
--self_denoising_lr=1.0

so, is there any mistakes in my test setting? or my training is not sufficient? Can you upload the best model of imagenet64 and 128? (since there is only the checkpoint of afhq)

[Shuangfei]

Hi, can you share your training script, also what’s the final loss value you get? And sorry that we cannot share the imagenet checkpoints due to licensing issue.

[Doraemonzzz]

We provide the training curve of cond_imagenet_64, with results as follows:

[Doraemonzzz]

We also use the following training script:

 torchrun   $code_dir/train.py \
        --config=${config} \
        --logdir=${logdir} \
        --data=${data} \
        --wandb \
        --wandb_project=${wandb_project} \
        --wandb_exp_name=${wandb_exp_name}

Specifically, I put the config in yaml, and the config for cond_imagenet_64 is as follows:

img_size: 64
channel_size: 3
patch_size: 2
channels: 768
blocks: 8
layers_per_block: 8
noise_std: 0.05
batch_size: 256
epochs: 200
lr: 1e-4
nvp: true
cfg: 0
drop_label: 0.1
sample_freq: 5
logdir: runs/imagenet64-cond

[Shuangfei]

Thanks. The training curves look pretty normal, so is the online FID curve, which means your checkpoint should be reasonable. I'm not exactly sure what causes the difference but 6.27 is way too high. Maybe try a smaller cfg weight, eg 2.1 instead of 2.3?

[Doraemonzzz]

Thank you, we will try other cfg for testing.

[encoreus]

I firstly searched for the cfg and got the outcome:

Then I kept the best cfg and searched for the denosing learning rate:

It seems that for imagenet64cond, the model I trained reach a limitation at FID 3.93, is this a reasonable value？ Or what other hyper parameter can I tune?

[Shuangfei]

Ok this is strange. The optimal cfg weight you found is too small, and the best FID is way too high (and you should be able to achieve this number with even a much smaller model.) The Imagenet128 result looks more reasonable though. Did you double check on the FID real stats and make sure it's correctly computed? The other place that could go wrong is by accidentally setting the annealed_guidance flag to False in here https://github.com/apple/ml-tarflow/blob/main/evaluate_fid.py#L77, which will result in the optimal cfg being smaller and also giving worse numbers.

[encoreus]

Thanks, I search the cfg and denoise_lr again and match the outcome of imagenet128cond to your paper, but the imagenet64cond is still strange, and I plan to retrain the model.
By the way, when I tune the hypter parameters of imagenet64uncond, I found that denoise_lr is effective but cfg and attn_temp seems valid, so when you get the final fid 18.42 of img64uncond, did you tune all three parameters (cfg, attn_temp, denoise_lr)?

[encoreus]

[Shuangfei]

Sounds good. If you want to retrain, maybe also consider the patch size 4 model in the paper which is faster to converge. For the unconditional model, CFG and attn_temp needs to be jointly searched, with the best cfg and attm_temp sometimes you are able to get slightly better FID by using a denoising_lr that’s slightly less than 1.

[encoreus]

Thanks!

[encoreus]

Excuse me, could you tell me the range of fid of AFHQ model provided in your checkpoint? I use your checkpoint and yeild a fid about 13.60.

[Doraemonzzz]

We trained our model on ImageNet 64x64 with a patch size of 4, and the loss and FID (Fréchet Inception Distance) metrics are shown in the figure below:

Is the loss and FID values are reasonable?

[Doraemonzzz]

On the other hand, we tested this model, and the results are as follows. Even our best results still have a gap compared to the results reported in the paper.

[Shuangfei]

Thanks for the updates. One thing I just realized the paper forgot to include is that the patch 4, 8x8 depth, 1024 width setting was trained for 300 epochs and a batch size of 480, instead of 200 epochs and batch size 256, and all other hyper parameters are the same as the patch 2 run. Checking our internal run's result, it has a comparable training loss to the one you posted above, but has slightly lower online FID. The FID results for this one is 3.99 with cfg of 1.7 (and we didn't tune the denoising lr). But in general cfg values between [1.3, 2.4] all yield FID below 4.5.

To make sure there is no unexpected issue, I'll also run a test with the exact patch 4 setting and train for 200 epochs and batch size 256 to see if I get the same result.