RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction
2026-08-10 • Sound
SoundArtificial Intelligence
AI summaryⓘ
The authors address a problem in turning brain scans into sounds, where using a pre-trained audio generator often creates realistic but incorrect sounds. They introduce RAG-Audio, a method that first decodes brain activity into a sound-related code, then finds a matching real audio example to help guide the generator. This approach improves the ability to identify the correct sound from brain data and reduces errors compared to direct generation. Their experiments show that starting the generator with a real example helps avoid mistakes caused by weak brain signals. Overall, the authors demonstrate that combining retrieval of real sounds with generation can improve brain-to-audio reconstruction.
brain-to-audio reconstructionfMRI decodingsemantic audio embeddingretrieval-augmented generationAudioLDMFréchet Audio Distanceprior dominationlatent trajectory initializationnearest-neighbor retrievalautoregressive model
Authors
Ambuj Mehrish, Sebastiano Vascon
Abstract
Brain-to-audio reconstruction is limited by \emph{prior domination}: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from $0.14$--$0.18$ for direct generation, near the $0.10$ chance level, to $0.40$--$0.43$, comparable to retrieval. It also reduces Fréchet Audio Distance by roughly an order of magnitude, from $13.49$ to $1.25$ for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.