Brain to speech decoders share neuron groups across speech forms
Mechanistic Interpretability Reveals Shared Causal Subspaces in Brain-to-Speech Decoders
Machine Learning
Summary
Decoding hidden speech like imagined or mimed speech from brain signals is much harder than decoding spoken words. The authors studied which parts inside a brain-to-speech decoder carry information that works across different kinds of speech. They found that small groups of neurons, especially from spoken speech input, help improve decoding of hidden speech. These neuron groups overlap in later stages of decoding, suggesting future decoders can be more efficient by focusing on shared representations from spoken speech data.
What this means in practice
- •For speech technology developers: Design brain-to-speech decoders that improve covert speech recognition by leveraging shared neuron groups found in vocalized speech data.
- •For neuroprosthetics engineers: Build more effective communication aids by focusing on later-stage shared neural representations to decode imagined speech signals.
Authors
Maryam Maghsoudi, Ayushi Mishra, Sanghamitra Dutta
Abstract
Decoding covert speech, such as mimed or imagined, from brain activity is harder than decoding vocalized speech. Cross-modal transfer, where information from one speech form helps decode another, is a promising remedy; yet how a decoder internally represents and processes brain activity from different speech forms remains unclear. In this work, we ask: which internal neurons of a decoder carry cross-modal information, and are these neurons shared across different speech forms? To answer these questions, we leverage mechanistic interpretability, using recordings of the same sentences in vocalized, mimed, and imagined input pairs for activation patching. We insert the decoder's internal activity for a sentence in one condition into its processing of the same sentence in another and measure the change in decoding accuracy. We find that no single neuron drives this benefit; instead, it arises from small groups of neurons, with vocalized speech as the most useful source. These groups are largely condition-specific in the early stage of the decoder but overlap in the later stage. These findings point toward more data-efficient covert speech decoders through training objectives that encourage shared later-stage representations learned mainly from vocalized data.