EEG captioner improves understanding of brain wave language translation
EEG-Captioner: Grounding EEG-to-Language Generation in Fundamental Signal Properties
Machine Learning
Summary
Translating brain signals into human language is tricky because creating fluent text doesn’t guarantee understanding the brain data itself. The authors created a large dataset linking EEG signals with detailed questions about their properties and captions grounded in the features of those signals. They designed a new model that processes both time-based and frequency-based EEG data and trained it to both answer questions about the signals and generate accurate captions. This approach improves the model’s ability to understand EEG signals and produce factually consistent descriptions.
What this means in practice
- •For brain-computer interface developers: Improve EEG interpretation in brain-computer interfaces by using models trained on detailed EEG property questions and grounded captions to increase signal understanding.
- •For clinical neurodiagnostic teams: Assist in interpreting complex EEG signals through technology that links signal properties to descriptive language, aiding clinical EEG analysis.
Authors
Leyuan Jin, Jiahao Mei, Yixuan Yuan, Shimeng Li, Mengyue Wu, Yue Ding
Abstract
Electroencephalography (EEG)-to-language models offer a promising approach for translating complex neural signals into human-readable descriptions. However, generating plausible language does not necessarily demonstrate that a model has learned to interpret the underlying EEG signal. A direct way to assess such interpretation is to probe objectively measurable properties of the signal itself. We therefore construct an EEG-language dataset of 160,781 segments annotated with signal-level question answering (QA) pairs, where each question targets one of five property families and requires a quantitative comparison involving spectral, spatial, temporal, or lateral signal properties. Each segment is also paired with a feature-grounded EEG caption. We further propose a Dual-Domain EEG Encoder that jointly models temporal waveforms and spectral-spatial representations, and introduce EEG-Captioner, a unified EEG-to-language framework trained with both signal-level QA and EEG caption supervision. Experiments show that the proposed encoder achieves the highest overall macro-averaged balanced accuracy (Macro-BA) among the compared EEG encoders, while EEG-Captioner achieves a Macro-BA of 88.81% on signal-level QA and improves caption factual consistency from 82.06% to 83.46% over caption-only language adaptation, demonstrating the complementary benefits of signal-level QA and caption supervision.