Pixel-Space Diffusion Transformers
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors explain that traditional image generation models compress images, which can lose fine details, and train parts separately, causing mismatches. To fix this, they explore modeling images directly in pixel space, which helps keep more detail and allows end-to-end training but comes with new challenges like handling high-dimensional data and designing efficient models. They review how Pixel-Space Diffusion Transformers work and discuss ways to unify image generation and understanding with text in one system. Their paper highlights current methods, challenges, and future research directions in making better vision models.
Latent Diffusion ModelsVariational Autoencoder (VAE)Pixel-space DiffusionTransformerHigh-fidelity Image GenerationDenoisingMultimodal ModelingTokenizationNoise SchedulingEnd-to-end Training
Authors
Renye Yan, Jikang Cheng, You Wu, Ling Liang, Wei Peng, Athanasios V. Vasilakos, Qingyu Zhao, Yu Zhang, Ehsan Adeli, Kilian M. Pohl, Guoying Zhao
Abstract
Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.