Adversarial training improves fine details in pixel image generation
Adversarial Training for Pixel Diffusion
Computer Vision and Pattern Recognition
Summary
Pixel diffusion models can create images by predicting pixels directly but often miss fine details seen in real photos. The authors show that adding adversarial training after initial training helps these models produce sharper and more natural textures without changing how they generate images. This method fixes missing high-frequency details important for realism, unlike other approaches that add noise or lose alignment with the image description. The improvement works well when the model directly outputs the image pixels, but not when using compressed forms.
What this means in practice
- •For graphic designers: Enhance image generation tools to produce images with sharper textures and more faithful details from text prompts.
- •For video game developers: Improve realistic environment textures by generating detailed images directly with higher fidelity and natural high-frequency details.
Authors
Xin Lin, Zhifei Zhang, Yuqian Zhou, Haitian Zheng, Zhe Lin, Ming-Hsuan Yang, Truong Nguyen
Abstract
Pixel diffusion models generate RGB images directly, avoiding the bottleneck of an autoencoder, yet their outputs still systematically underrepresent fine-scale natural-image statistics. We show that adversarial learning provides an effective post-training correction for this deficiency. Starting from a pretrained model, we retain its original diffusion or flow-matching objective and add an adversarial loss to the predicted output at non-high-noise timesteps, leaving the model architecture and sampling procedure unchanged. To our knowledge, this is the first systematic study of adversarial post-training for pixel diffusion. Across two pixel backbones, the method jointly improves distribution fidelity, coverage, prompt alignment, and perceptual quality. We further investigate why it works. Frequency-band and power-law analyses show that the original models systematically underproduce natural-image high-frequency content, while adversarial post-training restores this missing spectral power. In contrast, perceptual loss also increases high-frequency content but sacrifices distribution fidelity and prompt alignment. Nearest-neighbor, recall, and matched no-GAN SFT controls further rule out memorization, mode dropping, and additional optimization as simple explanations. Finally, we examine the boundary of this effect. Under the tested latent diffusion configurations, the same procedure does not produce comparable joint gains and adds almost no decoded high-frequency power. These results identify direct output access to the image statistics being corrected as a key factor governing when adversarial post-training succeeds.