Pixel diffusion model improves image quality using pretrained representations
PixelDiT2: Representation-Grounded Pixel Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Creating detailed images by gradually refining pixels usually lags behind methods that work on simpler, compressed image versions. The authors found this is because pixel-based methods must learn how to clean up noisy pixels and understand image features at the same time. Their new model, PixelDiT2, uses a powerful pretrained vision model to guide the pixel cleanup, letting the system focus on generating better images. This approach led to better and faster improvements on standard image tests.
What this means in practice
- •For computer vision engineers: Improve pixel-level image generation systems by incorporating pretrained vision model guidance for faster training and higher final quality.
- •For graphics software developers: Develop more accurate image synthesis tools that produce high-resolution pictures directly in pixel space without relying on autoencoders.$Commercial implications: Enables selling advanced image editing and creation software with superior output quality using the proposed diffusion approach.
Authors
Yongsheng Yu, Wei Xiong, Yichen Sheng, Shiqiu Liu, Jiebo Luo
Abstract
Recent advances in pixel-space diffusion models have narrowed the image quality gap with latent-space diffusion, but still converge more slowly and lag behind in final image quality. We argue that a key reason is the lack of an explicit representation prior: unlike latent diffusion, which usually denoises in a compact and structured latent space, pixel diffusion needs to learn denoising-friendly representations and pixel generation simultaneously from raw RGB space. To address this problem, we propose PixelDiT2, an end-to-end pixel-space diffusion model designed to decouple representation learning from pixel generation without introducing an autoencoder or latent reconstruction bottleneck. We propose representation grounding that uses a frozen pretrained vision foundation model to provide explicit per-patch representation guidance throughout denoising, allowing the pixel diffusion transformer to focus more on pixel generation. On ImageNet-256x256, PixelDiT2 achieves an FID of 1.46 after 600 epochs; at 512x512 resolution, PixelDiT2 achieves an FID of 1.48 after 680 epochs.