Change of inference result if use init_empty_weights

[pft-JoeyYang]

Hi, I tried to introduce accelerate's init_empty_weights to speedup the model loading:

from accelerate import init_empty_weighs

self.ckpt_path = kwargs.get('model_path', os.path.join(self.model_dir, 'anytext_v2.0.ckpt'))
clip_path = os.path.join(self.model_dir, 'clip-vit-large-patch14')

# Use accelerate here
with init_empty_weights:
    self.model = create_model(cfg_path, cond_stage_path=clip_path, use_fp16=self.use_fp16)
if self.use_fp16:
    self.model = self.model.half()
self.load_weights()

Also adding assign=True in self.load_weights():

def load_weights(self):
    self.model.load_state_dict(load_state_dict(self.ckpt_path, location='cuda'), strict=False, assign=True)
    print('Original weights loaded!')

However, the results seemed to differ (maybe became poorer) if I changed the loading method:
(from left to right are source overlay, original result and empty_weight result repectively; with same seed)

Is this an expected behavior?
Thank you.