Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on making pictures from sounds, which is called audio-to-image (A2I) generation. They point out that existing methods are limited because current datasets don’t have enough high-quality, well-matching sound and image pairs. To fix this, they created A2I-Set, a big and detailed dataset with matching audio, images, and captions. They also made a new testing set with human help and built a model called AudioCanvas that learns from their dataset. Their experiments show AudioCanvas makes better and more accurate images from sounds than older methods.
audio-to-image generationcross-modal generationdatasetstext-to-image modelscross-modal alignmentA2I-SetAudioCanvasmulti-modal learningfine-tuningaudio-conditioned image generation
Authors
Dongxu Ge, Shansong Liu, Cheng Gong, Xiao-Lei Zhang, Chi Zhang, Xuelong Li
Abstract
As an important subfield of cross-modal generation, synthesizing static visual content in the form of images from audio, namely audio-to-image (A2I) generation, has attracted increasing research attention in recent years. Nevertheless, despite the remarkable visual quality of modern text-to-image (T2I) models, the performance of A2I remains fundamentally limited by traditional datasets, which often lack both high-fidelity images and precise cross-modal alignment. As a result, existing methods still struggle to achieve high-quality audio-to-image generation through finetuning strong T2I models, thereby constraining practical applications in this area. Motivated by this gap, we introduce A2I-Set, a unified, high-quality tri-modal dataset consisting of 323K paired audio, images, and detailed text captions, specifically designed for audio-visual research, including audio-conditioned image generation. Besides, we developed a new mixed-source test set for the A2I task through human supervision. We further propose an A2I model, AudioCanvas, fine-tuned on our A2I-Set. Experiments show that AudioCanvas achieves more visually expressive as well as cross-modal alignment results that generally outperforming existing approaches. Our dataset and source code are available at https://github.com/gdx012/A2I-Generation.