Multimodal models learn to decode brain signals from EEG images
Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Reading brain signals with computers is difficult because brain activity varies a lot depending on the task and person. The authors created BraVista, a method that turns brainwave data into pictures, letting big visual and language AI models learn to understand them together. This approach works well on different brain-reading tasks like sleep stages and emotions without needing special brain data training first. They also found that how EEG data is turned into images really matters for success. Tests show the model actually uses brain signal details instead of just guessing from pictures.
What this means in practice
- •For medical device developers: Design brain-monitoring tools that decode multiple EEG tasks using vision-language AI without extensive EEG-specific training.$Commercial implications: Enables advanced multi-task EEG decoding devices for hospitals and clinics using AI, improving diagnostics and patient monitoring capabilities.
- •For wearable technology engineers: Create wearable brain signal interpreters that apply visual-language models to detect emotions, cognitive load, or sleep stages reliably.
Authors
Parastoo Azizeddin, Omid Sharafi, Maryam M. Shanechi
Abstract
Learning EEG representations that generalize across cognitive tasks, subjects, and recording conditions remains a key challenge in electroencephalography (EEG) decoding. Recent advances in foundation models have improved EEG decoding performance, yet a fundamental open question remains: how to effectively interface neural signals with these models to enable multi-task learning across datasets. To investigate this question, we introduce BraVista, a visual-language framework that encodes multichannel EEG signals as structured images and enables multi-task learning through instruction-conditioned vision-language models (VLMs). Our approach relies on continued post-training of a general-domain VLM, leveraging its visual and linguistic priors to adapt to neural signals without a separate large-scale EEG-specific pretraining stage. We evaluate BraVista on four datasets spanning sleep staging, emotion recognition, cognitive workload classification, and abnormal EEG detection, showing strong performance across these tasks. Further analyses show that the choice of EEG-to-image representation is critical to performance. Moreover, through controlled perturbations of the EEG signal, we observe a gradual performance degradation under increasing noise, suggesting that the model relies on EEG-relevant information rather than superficial visual patterns. Together, these findings establish structured visual representations as an effective and scalable interface between neural signals and general-domain foundation models for unified multi-task EEG decoding.