Structural self-teaching improves image generation with complex prompts
Structural Self-Teaching for Compositional Generation in Unified Multimodal Models
Computer Vision and Pattern Recognition
Summary
Generating images from detailed descriptions is hard for current AI models because they often miss objects or place things incorrectly. The authors propose a method called Structural Self-Teaching that uses the model's own ability to understand images to guide how it creates them. By teaching the generation part of the model where each described object should go, without needing extra labels, the images better match the complex prompts. This method works during training without slowing down image creation later.
What this means in practice
- •For image generation developers: Improve the accuracy of image generation from complex and detailed text prompts without extra input or slowing down inference.
- •For digital content creators: Generate images that better match intricate scene descriptions to enhance design, advertising, and media workflows.
Authors
Jaeyoo Park, Bohyung Han
Abstract
We propose Structural Self-Teaching (SST), a novel framework that enhances compositional image generation in diffusion-based unified multimodal models (UMMs) by leveraging their internal understanding path as a source of structural supervision. The generation path of UMMs often struggles with dense compositional prompts, leading to missing objects, attribute binding failures, and spatial errors. In contrast, their understanding path localizes the same phrases in images that satisfy these prompts. SST converts this phrase-level grounding into training-time structural signals for the generation path. Specifically, we extract phrase-conditioned support masks from internal attention maps derived from the understanding path and employ them through two mechanisms. Phrase-local contrastive alignment uses these masks as pooling regions and aligns each generated phrase region with the same region of the target image. A structural scaffold builds a phrase-level layout from the same masks and modulates an intermediate spatial state of the generation path during training. The scaffold is phased out during training and removed at inference. SST requires no external structural annotations for training and no additional input at test time. Extensive experiments across multiple UMMs demonstrate that SST improves compositional fidelity over LoRA fine-tuning on the same data, with no inference-time cost. Project page is available at https://bellos1203.github.io/sst