Context-aware graph network improves emotion recognition from EEG and eye movements
Context-Aware Topological Distillation Graph Network for Asymmetric EEG-Eye Movement Emotion Recognition
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
The paper tackles how to better understand emotions by looking at brain signals (EEG) and eye movements together. Unlike previous methods that treated these signals as separate, the authors created a graph-based model that connects EEG electrode data while factoring in eye movements and past information to improve recognition. Their approach also works well even when eye movement data is missing or noisy. Tests showed their method is more accurate and consistent across different people and scenarios than earlier models.
What this means in practice
- •For mental health device makers: Develop more reliable emotion detection systems that integrate EEG and eye tracking even when eye data is partially missing or noisy.$Commercial implications: Enables sale of advanced emotion-sensing wearable devices with improved robustness for mental health monitoring.
- •For human-computer interaction teams: Create adaptive interfaces that respond to user emotions by combining brain signals and eye movement data with higher accuracy and stability.
Authors
Peijun Shi, Xiujin Zhu, Zhenguo Lao, Chee-Onn Chow, Wei Ru Wong, Zhen Liu, Xiaobing Jiang
Abstract
EEG-eye movement emotion recognition can support mental-health monitoring and adaptive interfaces, but most fusion models treat the two signals as parallel features, classify each window alone and degrade when eye data are lost. We propose the Context-Aware Topological Distillation Graph Network (CTD-GNet), in which EEG features form the nodes of an electrode graph, and eye movements and preceding windows modulate its edges, nodes and readout, the edges through electrode-specific prompts. Bidirectional distillation of the graph, node relations and predictions couples this multimodal teacher with a single-window EEG student, improving the teacher and aligning the two graphs 35-fold. In leave-one-subject-out tests on SEED and SEED-IV against 17 baselines, CTD-GNet has the highest mean accuracy, 84.91% and 76.17%, significantly above 9 and 6 of 13 eye-based baselines after Holm correction, the smallest cross-subject spread among multimodal models and the highest worst-subject accuracy. It is the most accurate model at every rate of missing eye data on SEED-IV, keeps 66.1% on SEED without eye data, more than any EEG-only model, and stays 9.3 and 11.9 points above all single-window fusion models under eye-movement noise of one standard deviation. On film clips unseen in training, it leads on SEED with 77.1%.