An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how to improve image generation models that work directly with pixels, rather than abstract latent features. They found that training these pixel-based models from scratch is slower than training latent models. To fix this, they propose first training in latent space, then shifting to pixel space for fine-tuning. By carefully choosing how to make this transition, they created pixel-based models that work as well or better than latent models and run much faster. Their work gives clear guidance for building better pixel-space image generators.
generative modelingpixel-space diffusion modelslatent spacepre-trainingdecoder architecturenoise scheduleweight initializationimage generationinference speed
Authors
Dengyang Jiang, Ruoyi Du, Zhennan Chen, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Xiangpeng Yang, Huanqia Cai, Aiming Hao, Yuming Jiang, Peng Gao, Harry Yang, Steven Hoi
Abstract
This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.