Differential attention improves emotion recognition from eeg and speech fusion
Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
Machine Learning
Summary
Recognizing emotions can be done by looking at brain signals (EEG) and speech, but simply combining these two sources often doesn’t work well because noise in brain signals interferes. The authors created EmoSpeechBrain, a system that first reduces noise in EEG by focusing on differences in attention patterns and then smartly combines brain and voice data. This method improves emotion recognition accuracy significantly compared to using either EEG or speech alone or previous fusion methods. Their work shows that cleaning EEG noise is key to successfully mixing brain and speech signals for understanding emotions.
What this means in practice
- •For affective computing developers: Build more accurate emotion recognition systems that combine brain activity and vocal cues by suppressing EEG noise before fusion.
- •For mental health monitoring teams: Deploy improved multimodal tools that better detect emotional states from EEG and speech for monitoring patient well-being.
Authors
Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen
Abstract
Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.