Hierarchical EEG alignment improves sentence-level thought decoding
Thought-deciphering via hierarchical EEG-to-text alignment
Machine Learning
Summary
People who struggle to communicate because of disabilities like locked-in syndrome or cerebral palsy need better ways to send messages. The authors worked on a new method that uses brain signals recorded with EEG to identify entire sentences a person is thinking about. Instead of focusing on single letters or words, their method looks at sentences as whole ideas, which helps capture more meaningful communication. They designed a system to match brain activity with possible sentences using context, making it easier to find the right sentence the person is thinking of.
What this means in practice
- •For assistive technology developers: Create more effective non-invasive communication devices that decode full sentences from EEG signals for people with severe speech disabilities.$Commercial implications: Enables commercial brain-to-text communication tools providing practical sentence-level decoding from affordable EEG devices.
- •For consumer eeg device makers: Incorporate hierarchical EEG-to-text models to enhance sentence recognition capabilities in lightweight, low-cost EEG headsets.
Authors
Xiaopeng Mao, Morten Mørup, Xin Xu, Sang Wan Lee, Sadasivan Puthusserypady
Abstract
Communication is important for our well-being, but it can be impaired by disabilities, such as locked-in syndrome (LIS), cerebral palsy (CP), and dysarthria. Non-invasive brain-to-text brain-computer interfaces (BCIs) offer a promising solution by providing an alternative communication pathway for people with communication disabilities. However, most recent breakthroughs have relied on large, cumbersome, and costly neuroimaging modalities, such as functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). Numerous studies have attempted decoding using the lighter and cheaper electroencephalography (EEG), but the results remain inferior compared to the other modalities. More importantly, we do not understand how these modalities are related to text at the letter, word, sentence, and higher semantic levels. Many prior studies have focused on decoding below the sentence level because it is considerably easier to design an experiment around phonemes, letters, and words. However, during communication, sentences play a much more important role than words and letters. This is because sentences carry information at such a high level that they can be understood even if some words and letters are missing. However, within sentence-level decoding, sentence classification is not as well investigated as sentence generation, making it difficult to decide whether a sentence should be retrieved or generated in the context of EEG-to-text decoding. This paper strives to investigate sentence-level EEG-to-text classification via a novel hierarchical alignment framework, where the main goal is to retrieve a read sentence using its context and EEG as inputs.