Multimodal system improves emotion recognition by adapting source reliability
ReH-FUSE: Reliability-Aware Hierarchical Fusion of Experts for Multimodal Emotion Recognition in Conversation
Machine Learning
Summary
Recognizing emotions in conversations is challenging because different sources like words and tone can be more or less reliable depending on the situation. The authors propose ReH-FUSE, a new system that decides when to trust words, sounds, or both together based on the conversation context. This approach lets the system focus on the most useful information for each moment. Their tests on standard emotion datasets show that this method works better than simply mixing all sources evenly.
What this means in practice
- •For voice assistant developers: Improve emotional understanding in speech-based assistants by dynamically balancing text and audio signals for better user response.
- •For customer service software teams: Enhance call center emotion detection by selectively weighing spoken words and vocal tone to better assess customer sentiment.
Authors
Guan-Hua Wen, Hou-Chiang Tseng, Kuan-Yu Chen
Abstract
Multimodal emotion recognition in conversation (ERC) requires adapting to the instance-dependent reliability of different evidence sources. Lexical content may be decisive, vocal expression may provide complementary cues, or accurate recognition may require cross-modal interaction; fixed fusion does not explicitly account for this variation. We propose ReH-FUSE, a reliability-aware framework with dialogue-aware text, audio, and cross-modal experts. Its decision-level router first models the relative preference between text and audio and then balances the resulting unimodal mixture against the cross-modal expert. This factorization separates unimodal competition from cross-modal selection. Across three independent runs on IEMOCAP, ReH-FUSE achieves 74.34% weighted F1 and 73.11% macro F1; on MELD, it achieves 68.03% weighted F1. Controlled ablations show that learned routing outperforms uniform expert averaging and benefits from cross-modal interaction.