[Question]: LayerDiffuse Fails to Generate Transparent Images with Re-trained SDXL Models (e.g., NOOBAI XL, animagin XL 4.0, Kohaku Zeta)

[linfen0]

What happened?

LayerDiffuse works correctly with the original workflow-provided models. However, when using re-trained SDXL models (such as NOOBAI XL, animagin XL 4.0, Kohaku Zeta—primarily anime-style models), LayerDiffuse either generates images with solid color backgrounds or fails to work entirely.
Fails to generate transparent img

Solid color background

Transparent images generated（A realistic style model）

Steps to reproduce the problem

Load those SDXL model (e.g., NOOBAI XL, v-pred,animagin XL 4.0, Kohaku Zeta) into the workflow.
Enable LayerDiffuse to generate transparent images.
Observe the output.

What should have happened?

generate transparent images as it does with the original models.

Commit where the problem happens

master branch

Sysinfo

win11

Console logs

Currently, no console errors are observed, but if logs are necessary for debugging, please let me know, and I’ll provide them.

Workflow json file

The example workflow

Additional information

[linfen0]

The behavior observed with these re-trained SDXL models might be related to the inherent sensitivity of the latent space, which could explain why LayerDiffuse struggles in these cases.

I thought this phenomenon could potentially be interesting for your research, perhaps as a way to explore how different degrees of model modification affect the functionality of tools like LayerDiffuse. For instance, it might help in understanding the threshold at which these tools become incompatible.

[Bocian-1]

I've made a workflow that can go around this a bit by generating an image with any checkpoint using ControlNet images extracted from a compatible checkpoint's generation and then crops it with the Layer Diffusion's mask. It's an extra step but allows to make transparent images with any architecture that doesn't have a trained Layer Diffusion model yet. Quality is a bit worse since the crop may still not match the generation perfectly despite the ControlNets and there are potential issues with Lora, but as far as the crop quality itself goes it is definitely better than using a background removal node.
https://civitai.com/models/1461145?modelVersionId=1652336

[fappaz]

The behavior observed with these re-trained SDXL models might be related to the inherent sensitivity of the latent space, which could explain why LayerDiffuse struggles in these cases.

I thought this phenomenon could potentially be interesting for your research, perhaps as a way to explore how different degrees of model modification affect the functionality of tools like LayerDiffuse. For instance, it might help in understanding the threshold at which these tools become incompatible.

I think you're better off posting this in the actual repository for the models, since this one is just the ComfyUI wrapper.

[fappaz]

@linfen0 Install Fooocus_Nodes in your ComfyUI and try again. I don't even need to use any of their nodes, just having that package active is enough to "fix" SDXL Attention Injection in my case.

I've been trying to debug this issue for quite some time now but am not familiar with the code - it seems they monkey patch the models, and that somehow makes the LoRA work better.

Perhaps @huchenlei could chip in 🙏