Autoregressive pretraining improves text to image generation quality and efficiency
Know, then Draw: A New Learning Paradigm for Visual Generation
Computer Vision and Pattern Recognition
Summary
Creating detailed images from text prompts needs both knowing what to draw and how to draw it well. This paper shows that training a computer model first to understand the high-level meaning using a step-by-step prediction method (autoregression) helps it learn better and faster. Then, a second step called diffusion is used to create high-quality images precisely. The authors show that focusing on semantic understanding is more important than just how good the early images look. They trained a large text-to-image model efficiently and with good results, proving this two-step approach works.
What this means in practice
- •For graphic design teams: Develop text-to-image tools that produce more accurate and detailed images with lower training cost using a two-stage training method combining autoregression and diffusion.
- •For game developers: Improve in-game content generation by using models trained to better understand scene semantics before rendering detailed visuals, reducing compute needs.
- •For advertising agencies: Create customized, high-quality images from descriptions faster and cheaper by adopting this pretraining and rendering pipeline in AI image generation services.$Commercial implications: Enables sale of efficient AI-powered image generation platforms for custom advertising content creation.
Authors
Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, Hanwen Jiang
Abstract
Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.