Text to image diffusion models improve attribute and spatial accuracy
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Computer Vision and Pattern RecognitionMachine Learning
Summary
Text-to-image AI models often make mistakes when asked to create pictures from descriptions with multiple objects or specific details, like mixing up attributes or reversing spatial relations. The authors show that this happens because the way the text is understood doesn't align well with how images are generated. They develop a method called ORCA that connects the text and image understanding better during training, helping the models create more accurate images without slowing down the creation process. The result is clearer, more correct pictures from complex text prompts.
What this means in practice
- •For ai model developers: Improve training of text-to-image diffusion models to reduce errors in multi-object scenes and attribute placements.
- •For creative ai service providers: Enhance image generation accuracy from text for applications in digital art and content creation platforms.$Commercial implications: Enables more reliable text-to-image tools that produce correct multi-object and attribute-based images, enhancing products for artists and marketers.
Authors
Arshia Hemmat, Amirhossein Vahidi, Amitis Shidani, Mohammad Vali Sanian, Hesam Asadollahzadeh, Aryan Yazdan Parast, Mohammad Lotfollahi
Abstract
Text-to-image diffusion models fail predictably on compositional prompts: attributes bind to the wrong objects, spatial relations invert, and multi-object scenes lose count. Recent architectures already augment CLIP with a T5 encoder precisely because CLIP's contrastive embedding loses compositional structure, yet these failures persist. We argue the binding problem is therefore not one of missing information but of misaligned information: a text encoder preserves compositional structure, but in a representation space shaped by language modelling rather than vision, and the denoising objective does not directly reward aligning the two. We show this correspondence can be supplied as an explicit training signal, that the relevant cross-modal information is concentrated in a low-rank subspace of self-supervised visual features, and that supplying it can be folded into diffusion training as a single auxiliary loss. Our method, ORCA (Orthogonal Residual Compositional Alignment), aligns the latent of a diffusion transformer with a low-rank target derived from a frozen visual encoder, through a predictor whose orthogonal basis is parameterised by a learned residual between T5 and CLIP embeddings, which provides a prompt-dependent signal for selecting the visual readout subspace. We prove that the cross-modal information recoverable at a given rank is bounded by the spectral mass of the visual encoder's covariance in the top components. Across three diffusion-transformer backbones (DiT-B/2, DiT-L/2, U-ViT-L), ORCA improves FID and GenEval over both vanilla and REPA baselines at zero inference-time cost; on DiT-L/2 it reaches FID 16.65 and GenEval 0.291 at 200K steps, exceeding the strongest 400K baseline at half the training cost, with the largest gains concentrated on attribute binding, spatial relations, and multi-object prompts.