Feature: Add Z-Image-Turbo regional guidance #8672
Open
+24,511
−5,347
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
api
python
Collaborator
|
I will start my review after the base z-image-turbo support is merged, which should be soon. |
Collaborator
|
Where are we on this one? Now that the primary Z Image and Controlnet PR's ae in, this should be pretty straight forward? Can you rebase this please? I'll run through the code and tests. |
Pfannkuchensack
force-pushed
the
feat/z-image-regional-guidance
branch
from
f4929ae to
56c7fbc
Compare
Implements regional prompting for Z-Image (S3-DiT Transformer) allowing different prompts to affect different image regions using attention masks. Backend changes: - Add ZImageRegionalPromptingExtension for mask preparation - Add ZImageTextConditioning and ZImageRegionalTextConditioning data classes - Patch transformer forward to inject 4D regional attention masks - Use additive float mask (0.0 attend, -inf block) in bfloat16 for compatibility - Alternate regional/full attention layers for global coherence Frontend changes: - Update buildZImageGraph to support regional conditioning collectors - Update addRegions to create z_image_text_encoder nodes for regions - Update addZImageLoRAs to handle optional negCond when guidance_scale=0 - Add Z-Image validation (no IP adapters, no autoNegative)
Fix windows path again
Pfannkuchensack
force-pushed
the
feat/z-image-regional-guidance
branch
from
56c7fbc to
1a37af3
Compare
Pfannkuchensack
requested review from
blessedcoolant and
lstein
as code owners
Changed the guidance_scale check from > 0 to > 1 for Z-Image models. Since Z-Image uses guidance_scale=1.0 as "no CFG" (matching FLUX convention), negative conditioning should only be created when guidance_scale > 1.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds Regional Guidance support for Z-Image (S3-DiT Transformer) models, enabling users to apply different prompts to different regions of the image using attention masks.
Key implementation details:
Backend:
ZImageRegionalPromptingExtensionclass that builds regional attention masksZImageTextConditioningandZImageRegionalTextConditioningdataclasses for managing regional text embeddingspatch_transformer_for_regional_promptingcontext manager0.0= attend,-inf= block) inbfloat16dtype[img_tokens, txt_tokens](different from FLUX's[txt_tokens, img_tokens])Frontend:
buildZImageGraph.tsto support regional conditioning collectorsaddRegions.tsto createz_image_text_encodernodes for Z-Image regionsaddZImageLoRAs.tsto handle optionalnegCondwhenguidance_scale=0validators.ts(no IP adapters, no autoNegative support)guidance_scale > 0Related Issues / Discussions
#8670
Extends Z-Image support (from the Z-Image-Turbo PR) with regional prompting capabilities.
QA Instructions
Merge Plan
Should be merged after the main Z-Image support PR (
feat/z-image-turbo-support), as this builds on top of that implementation.Checklist
What's Newcopy (if doing a release after this PR)