FMRI brain signals decode natural speech using multimodal time models
Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment
Machine LearningArtificial Intelligence
Summary
Figuring out what speech a person hears from brain scans is very difficult because brain signals change slowly and can be noisy. The authors developed NOEMA, a system that links brain activity patterns to speech segments by combining text, sounds, and speech features over time. Tested on two real speech datasets, NOEMA could pick the right sentence from thousands of choices more than 42% of the time, much better than previous methods. This shows brain scans contain detailed info about natural speech that can be decoded using clever time-based and multi-sense models.
What this means in practice
- •For brain computer interface teams: Enhance speech decoding accuracy from fMRI signals by integrating temporal context and multiple speech features to improve assistive communication devices.
- •For audio signal processing teams: Use combined text, audio, and speech embeddings aligned to brain signals to develop better models linking human perception and audio features.
Authors
Corentin Caris, Louis Jalouzot, Nona Rajabi, Géraud Ilinca, Ahmed Hassayoune, Asilbek Muminov, Thaddée Delebarre, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual, Antoine Collas
Abstract
Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.