Cross brain scans improve speech decoding across different people
Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings
SoundArtificial Intelligence
Summary
Decoding what speech a person hears from brain signals is hard because brain activity is complex and varies between people. The authors combine two types of brain scans, fMRI and MEG, to better understand brain signals related to hearing speech and create models that work well across many people. Their method not only improves accuracy in identifying perceived speech but also requires less training time. They also visualized the brain regions involved to confirm their approach works effectively.
What this means in practice
- •For medical device developers: Design brain-computer interfaces that decode heard speech signals more accurately and efficiently across different users.$Commercial implications: Enables creation of commercial non-invasive speech decoding devices usable by many patients without expensive retraining per user.
- •For neurotechnology engineers: Build neural decoding tools that fuse different brain scanning methods for improved speech recognition in brain data.
Authors
Aoke Zhang, Jing Chen
Abstract
Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.