Summary
Recognizing human emotions from brain and body signals is tricky because brain activity changes in many ways and different devices see different things. The authors created a computer model named GMSI-Net that looks at brain waves over time, frequency, and brain regions, while also combining other body signals to capture emotions better. The model carefully keeps unique details from each type of signal while also finding shared emotion clues. Tests show it can recognize emotional states like valence and arousal with high accuracy on two well-known datasets. This approach shows that combining detailed brain data with other body signals helps machines understand emotions better.
What this means in practice
- •For affective computing developers: Build improved emotion recognition systems by integrating detailed EEG features with other physiological signals for better accuracy in detecting user emotions.
- •For health monitoring teams: Enhance real-time stress and mood assessment tools by using combined EEG and peripheral signals analyzed through multiscale learned features.
Authors
Jian Zhang, Manrong Liu, Yilei Huang, Wenhua Liu, Xiaohui Yang, Lijun Yang
Abstract
Multimodal physiological signals provide complementary cues for emotion recognition. They also raise two related challenges. Emotion-related EEG patterns vary across time, frequency bands, electrodes, and brain regions. Different physiological modalities also contain both modality-specific and shared affective information. This paper presents GMSI-Net, an EEG-centered multimodal emotion recognition framework. It has three key components. The temporal-spatial-spectral-region (TSSR) encoder models multiscale temporal-channel patterns, differential entropy features, and local-to-global brain-region relations. The modality-specific representation (MSR) branch uses bidirectional cross-modal attention and learnable gates. It controls information transfer while retaining modality-specific cues. The modality-invariant representation (MIR) branch learns a shared affective representation. An adaptively weighted contrastive loss aligns this representation across modalities. Independent multiscale Conv-Transformer encoders process the peripheral physiological signals. GMSI-Net is evaluated on DEAP and MAHNOB-HCI using within-subject random ten-fold cross-validation. The model reaches 96.21 (1.47) and 96.41 (1.60) for valence and arousal on DEAP. It reaches 94.10 (2.86) and 94.48 (3.77) on MAHNOB-HCI. These results indicate that combining EEG features with complementary physiological information can improve emotion recognition.