Brain activity data helps decode natural speech from podcasts
Cephalonauts One: A deep fMRI dataset for decoding naturalistic speech in the human brain
Machine LearningArtificial Intelligence
Summary
Understanding how our brains process everyday speech is challenging because brain signals are complex and data is limited. The authors collected a very large set of brain scans from people listening to podcasts in their own language. They matched brain activity with the exact podcast audio and transcripts to try to identify which spoken segment someone was hearing just from their brain scan. They also tested computer methods to decode this from the brain data and found that more data helps improve decoding accuracy.
What this means in practice
- •For brain-computer interface developers: Train systems to match brain signals with specific speech sounds for communication aids or prosthetics.$Commercial implications: Enables speech decoding devices that convert brain activity into spoken words for people who cannot speak.
- •For audio technology engineers: Create tools that align neural responses with audio content to improve personalized sound or hearing devices.
Authors
Antoine Collas, Louis Jalouzot, Géraud Ilinca, Corentin Caris, Romain Valabrègue, Ahmed Hassayoune, David Goncalves, Madeleine Hueber, Thaddée Delebarre, Julien Savatovsky, Clara Fonteneau, Charles Maussion, Bertrand Thirion, Alexis Thual
Abstract
Cephalonauts One is a whole-brain 3 Tesla (3T) functional magnetic resonance imaging (fMRI) dataset recorded while subjects listened to audio podcasts. Three healthy subjects underwent multiple scanning sessions, each consisting of five 15-minute runs, while listening to podcasts in their native language. With 30 hours of fMRI data per subject, the current release is the deepest available fMRI dataset using naturalistic speech stimuli. The dataset pairs brain activity with the corresponding podcast audio, transcript annotations, and derived stimulus embeddings. Furthermore, we introduce a brain decoding benchmark formulated as audio segment retrieval: given fMRI activity from a held-out session, the decoder must identify the corresponding time-aligned podcast audio segment among candidate segments. We provide standardized splits, evaluation metrics, and baseline decoders for this task. Finally, a scaling analysis shows that decoding performance improves continuously with the amount of training data per subject.