Multimodal domain adaptation improves eeg eye emotion recognition
IDEAL: A Multimodal Domain Adaptation Framework for EEG-Eye Emotion Recognition
Human-Computer Interaction
Summary
Recognizing human emotions from brain signals (EEG) and eye movements is tricky because people’s physiological responses vary a lot. Existing methods typically focus on one type of signal and try to align features, but they miss important differences at the raw data level and across multiple signals. The authors propose IDEAL, a method that expands and selects confident data instances and then aligns features across different signals carefully, preserving emotional distinctions. Their experiments show this method is better at recognizing emotions across different people and datasets.
What this means in practice
- •For human-computer interaction teams: Improve emotion recognition accuracy in systems by adapting EEG and eye movement signals across different users.
- •For audiovisual content developers: Enhance emotional feedback based on synchronized brain and eye data for interactive media or games.
Authors
Yang Wu, Jinpeng Li
Abstract
Electroencephalography (EEG) emotion recognition serves as a pivotal interface for human-computer interaction, yet the physiological variability across individuals complicates the already challenging task of fusing heterogeneous physiological signals (e.g., EEG and eye movements). However, most prevalent domain adaptation paradigms are tailored for unimodal scenarios, failing to address the heterogeneity of multimodal signals. Furthermore, they predominantly rely on feature-level alignment, overlooking the fundamental data-level discrepancy, which risks compromising fine-grained discriminative information during aggressive adaptation. To bridge these coupled gaps, we propose Instance-based Domain Expansion and Adversarial Learning (IDEAL), a unified framework that synergizes instance-level curriculum expansion with feature-level hierarchical adversarial alignment. IDEAL first introduces a multi-model collaborative screening mechanism, which propagates high-confidence target samples to explicitly bridge the distributional gap at the data level via a quantity-quality equilibrium strategy. We provide a theoretical analysis that this instance expansion strategy strictly tightens the upper bound of the target risk. Subsequently, a hierarchical adversarial network, augmented with the angular-contrastive constraints, progressively aligns representations from low-level statistics to high-level semantics while preserving class separability. Extensive experiments on four benchmark datasets demonstrate that IDEAL significantly outperforms state-of-the-art methods. To facilitate reproducibility and future research, our source code is publicly available at https://github.com/WY-BCI-Club/IDEAL.