Non-invasive brain recordings decode sentence meaning into text
The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding
Computation and LanguageMachine Learning
Summary
Decoding spoken words from brain signals is usually hard because these signals are noisy. The authors found that targeting the overall meaning of sentences, rather than individual sounds or words, works better for non-invasive brain recordings. They created a system that first translates brain signals into a kind of meaning representation, then turns that into text. This approach helps recover what was being thought more accurately without needing exact word-by-word matching.
What this means in practice
- •For assistive technology developers: Convert brain signals of users unable to speak into meaningful text based on sentence-level meanings rather than words.
- •For speech interface designers: Create brain-controlled devices that interpret intended sentence meanings for communication without relying on precise word decoding.
Authors
Gilad D. Landau, Dulhan Jayalath, Oiwi Parker Jones
Abstract
Non-invasive speech decoding remains constrained by the low signal-to-noise ratio of neural recordings, which makes fine-grained reconstruction of phonemes or individual words difficult. Motivated by neuroscientific evidence that high-level semantic representations are distributed across cortical regions and evolve over slower temporal scales, we hypothesize that semantic content may provide a more suitable target for non-invasive decoding than low-level acoustic or lexical features. We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space. Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language. This semantic bottleneck enables recovery of high-level meaning without word-level alignment. We describe the core principles of the approach, its implementation, and the strategies used to mitigate the challenges of learning a reliable neural-to-semantic mapping. Finally, we compare against prior non-invasive Brain2Text methods and show improved sentence-level results.