Improving image generation by guiding pixel diffusion transformers
Learning Representations to Guide Pixel Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Generating images with computers involves turning abstract ideas into pixels. The authors found that making image representations better for understanding doesn’t always help when creating images, because some important details get lost along the way. They propose a new way called Representation Guidance that shapes and uses internal image information better during image creation. This method helps the system generate clearer and more accurate images, even when scaling up in size or adding text descriptions.
What this means in practice
- •For image generation teams: Generate high-quality images more efficiently using improved pixel diffusion techniques with compact internal guidance.
- •For text-to-image developers: Enhance text-to-image models by integrating compact representation guidance to better capture semantic and spatial details.
- •For graphic design software developers: Integrate efficient generative techniques to offer clearer and faster image synthesis tools in creative applications.$Commercial implications: Enables creation of advanced image synthesis features that improve graphic design products’ quality and speed.
Authors
Zihao Pan, Mengting Chen, Cheng Shi, Taihang Hu, Zhengrong Yue, Zikang Wang, Yuan Wang, Wei Li, Sibei Yang
Abstract
Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as externally aligned semantics can be progressively lost during pixel synthesis. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling, without necessarily yielding representations better suited to generation. These insights suggest that a useful generative representation must be both properly formed and actively exploited. Motivated by these insights, we introduce Representation Guidance (RG), which directly forms and exploits internal representations while retaining a plain Transformer architecture. A low-dimensional bottleneck forces generation-relevant information into a compact representation, which is then explicitly used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset. RG learns clearer semantic and spatial structure while scaling efficiently in both parameters and computation. On ImageNet 256x256, RG-L/16 with only 459M parameters outperforms substantially larger pixel diffusion transformers, while RG-XL/16 achieves an FID of 1.55. RG further reaches an FID of 1.74 at 512x512 and extends to text-to-image generation, achieving 0.89 on GenEval and 87.1 on DPG-Bench.