About the training result of stage 1

[fffanli]

After the training of stage 1, model.safetensors and optimizer.bin and random_states have been saved.However,I don't get the ip.bin and style_aware_bin.So I use the following codes for saving necessary ckpt but I don't get style in the inference result which is terrible. I'm confused about how to save the correct training ckpt .

if accelerator.is_main_process:
torch.save(accelerator.unwrap_model(ip_adapter).state_dict(),
os.path.join(save_path, "ip.bin"))
torch.save(accelerator.unwrap_model(style_aware_encoder).state_dict(),
os.path.join(save_path, "style_aware_encoder.bin"))

[Jeoyal]

Hi @fffanli , thank you for your interest in our work, please see issues #59 and #69.

[fffanli]

Thank u for your advice!I've resolved that problem.Now when I test the result of training stage 1,the generated image doesn't get the style and I don't why .
test code:
ip_ckpt = "xx/results/checkpoint-110000/model.safetensors"
style_aware_encoder_path ="xx/results/checkpoint-110000/model_1.safetensors"
taining code:
CUDA_VISIBLE_DEVICES=5,6 accelerate launch --num_processes 2 --multi_gpu --mixed_precision "fp16"
tutorial_train_styleshot_stage_1.py
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5/"
--image_encoder_path="laion/CLIP-ViT-H-14-laion2B-s32B-b79K"
--image_json_file="xx/data/train.jsonl"
--image_root_path="xx/data"
--mixed_precision="fp16"
--resolution=512
--train_batch_size=4
--dataloader_num_workers=4
--learning_rate=1e-05
--weight_decay=0.01
--output_dir="xx/results"
--save_steps=1000
train.jsonl:
{"image_file": "wikiart/william-h_32.jpg", "content_prompt": "A plaza during rush hour with people walking quickly", "content_image_file": "content/square_A plaza during rush hour with people walking quickly.png"}
{"image_file": "wikiart/william-h_33.jpg", "content_prompt": "A plaza filled with autumn leaves and street performers", "content_image_file": "content/square_A plaza filled with autumn leaves and street performers.png"}

[Jeoyal]

Hi, what does your training images and results look like?

[fffanli]

Hi, what does your training images and results look like?
I think the content is correct but the style doesn't match.The prompt is "a cute cat"
Here is the output_train image:

The style image is as following:

[Jeoyal]

Hi, this style is a more specific and fine-grained style, I recommend you to train a simple style LoRA on more powerful diffusion model like Flux and Qwen-Image.

[fffanli]

this style is a more specific and fine-grained style, I recommend you to train a simple style LoRA on more powerful diffusion model like Flux and Qwen-Image.

Hello, thank you very much for your reply. The style_aware_encoder is trained in Stage 1. I used images from StyleBench to test the training results of Stage 1, but the style still failed to transfer correctly. I am wondering if it is still possible to proceed with Stage 2 training? I did not modify the training code of Stage 1, so why is the training effect not satisfactory? Or will the training effect be better in the second stage?

[Jeoyal]

What is the size and variety of your training dataset?

[fffanli]

What is the size and variety of your training dataset?

I use the wikiart deataset,about 80 thousand images.

[fffanli]

What is the size and variety of your training dataset?

I want to work on the field of style transfer. Could I have a discussion with you about it?

[Jeoyal]

Yes, feel free to email me.