Brain signals aligned with text enable imagined speech decoding
Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech
Machine Learning
Summary
Decoding what people imagine saying based on their brain signals is tricky because traditional methods treat each spoken idea as completely separate. The authors developed a new way to link brain activity to the meaning of words by training a system that matches brain patterns with text embeddings from a language model. This method performed as well as existing classifiers but also lets the system judge how confident it is and compare new imagined phrases to known meanings. Their tests involved multiple subjects and showed promises for more flexible and reliable brain-computer communication.
What this means in practice
- •For brain-computer interface developers: Improve imagined speech decoding systems by embedding EEG signals directly into a pretrained language space for better intent recognition and confidence scoring.
- •For assistive technology engineers: Build communication aids that retrieve semantically similar phrases from brain signals without relying on fixed label sets, allowing more naturalistic user interaction.
Authors
Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri, Luca Mainardi
Abstract
Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.