Dataset enables decoding images from brain activity induced by sound
AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Scientists created a new brain imaging dataset that records brain activity while people imagine pictures based on sounds, without showing any visuals. This helps separate brain signals from actual images versus those created inside the mind. They also set up ways to test how well computers can guess what people imagine from the brain scans. Early tests show current methods struggle, but some meaningful parts of the imagined scenes can be detected. This dataset will help improve brain-reading technology using sounds as triggers.
What this means in practice
- •For brain computer interface developers: Develop and test models that decode imagined visual scenes triggered by sound rather than visual cues using the dedicated AVID dataset.
- •For speech recognition engineers: Train systems that relate spoken language descriptions to brain-activated visual imagery to improve multimodal speech interfaces.
Authors
Shiqi Shen, Shurui Li, Xilin Zhang, Yuanning Li
Abstract
Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress is limited by the lack of dedicated, training-scale VMI datasets. Current VMI paradigms often rely on visual prompts or short-term memory images, introducing visual and mnemonic confounds that make it difficult to disentangle neural activity associated with endogenous image construction from signals driven by prior or concurrent visual input. Furthermore, existing VMI datasets offer limited descriptive control over individual visual attributes, making it difficult to systematically assess fine grained visual details. To address these gaps, we introduce AVID, the first dedicated training-scale 5.0 Tesla functional magnetic resonance imaging (fMRI) resource for auditory-induced VMI decoding. AVID comprises approximately 67 hours of dense recordings from nine participants. The dataset is structured into two complementary splits: a scene-level complex split using naturalistic Mandarin descriptions and a controlled simple split with explicitly specified visual attributes. We further formalize a standardized evaluation protocol that decouples source-image consistency from semantic alignment, allowing for a more nuanced assessment of imagery reconstruction. Baseline qualitative and quantitative results show that current reconstruction pipelines remain limited on AVID, although the most successful reconstructions preserve recognizable scene level semantics from spoken descriptions. By combining dedicated training-scale neural recordings and standardized evaluation resources, AVID provides both a data foundation for imagery-specific model development and a benchmark for evaluating VMI decoding with substantially reduced sensory contamination.