PixARG improves image generation speed and quality using pixel-level retrieval
PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generation
Computer Vision and Pattern Recognition
Summary
Diffusion models can generate realistic images but take a long time and sometimes make unrealistic mistakes. The authors of this paper created PixARG, which speeds up image creation by borrowing parts of images found by searching with your text description. PixARG smartly adds less noise where the search found matching details and more noise elsewhere, helping the model focus and finish faster. This approach also improves the look and relevance of the generated images by combining the search results with the generation process in a pixel-by-pixel way.
What this means in practice
- •For graphic design teams: Generate high-quality images faster by integrating retrieved real-world visual details tailored to text prompts, reducing computational cost.
- •For photo editing software developers: Incorporate adaptive noise modulation based on pixel relevance to accelerate and refine AI-driven image enhancements and editing features.
Authors
Zewei Xin, Qinya Li, Chaoyue Niu, Fan Wu
Abstract
While diffusion models achieve impressive image generation capabilities, they suffer from high computational costs, implausible artifacts, and limited adaptability to dynamic or private data due to static, centralized training. To address these, we propose PixARG, a novel retrieval-augmented generative framework that integrates dynamic real-world image retrieval into the diffusion pipeline to simultaneously enhance efficiency and visual realism. Specifically, PixARG constructs a pixel-level affinity map between the user's text prompt and the retrieved image to guide a heterogeneous noise initialization. By modulating noise intensity across spatial regions, this mechanism preserves semantically relevant structural priors while perturbing less relevant areas, thereby significantly accelerating convergence. To accommodate this spatially non-uniform noise and ensure synchronized generation, we introduce a pixel-wise asynchronous denoising scheduler and a spatiotemporal denoiser conditioned on the affinity map. Furthermore, we derive an affinity-adaptive training objective that elegantly balances deterministic restoration with stochastic regeneration across pixels. Experiments on LAION and COYO datasets across offline, online, and user-specific settings show PixARG enhances realism and semantic alignment while drastically reducing diffusion steps.