Role-aware framework improves emotion recognition in conversations
From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition
MultimediaComputer Vision and Pattern Recognition
Summary
Understanding how people feel in conversations is tricky, especially when only part of the interaction is visible. The authors created a system that uses videos of listeners and audio from speakers to better guess the listener’s emotions. Their approach handles unclear visual cues by using speaker context to help decide the listener’s feelings. This method showed a big improvement in recognizing emotions on a challenging test set.
What this means in practice
- •For call center teams: Detect listener emotions using only listener video and speaker audio to improve customer service quality in real-time interactions.
- •For mental health app developers: Incorporate emotion recognition that distinguishes subtle listener reactions from speaker context for better assessment of user mood.
Authors
Wei Wang, Zhaowu Li, Jianjie Luo, Fu Lee Wang, Lap-Kei Lee, Zhenguo Yang
Abstract
In this paper, we propose a Role-aware Stimulus-guided (RASG) framework for interlocutor emotion recognition, which predicts listener emotions from listener-only videos and speaker-only audios. RASG consists of Role-aware Visual Transfer (RVT) and Stimulus-guided Boundary Reasoning (SBR) modules, which address supervision mismatch due to the lack of labeled listener data and ambiguity among visually similar listener reactions whose interpretation depends on speaker context, respectively. More specifically, RVT selects speaker samples whose facial expressions support their emotion labels. It then filters listener tracks and uses reliable pseudo-labels to train a listener-centric visual expert. SBR uses a two-class language reasoner only when the visual model is uncertain. It treats speaker audio and text as context rather than direct emotion evidence to distinguish similar listener reactions. Experiments conducted on MER-Cross dataset shows that RASG achieves 76.25\% on MER-Cross and improves the performance of the baseline over 17\%. Our team ranks second in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026.