Multimodal model tracks group emotions every second in dialogues
Multimodal Temporal Modeling for Continuous Group Emotion Recognition in Multi-party Dialogues
Multimedia
Summary
Understanding how a group feels during a conversation can help make dialogue systems more natural. The authors created a method to recognize group emotions like mood and energy every second using both sound and video. They also introduced a new idea called the Mixed state, which shows when group members feel differently. Their approach works better than some existing models but finds it harder to detect emotions when the group’s feelings diverge a lot.
What this means in practice
- •For dialogue system developers: Create conversational agents that understand group mood dynamically in multi-person interactions for more natural responses.
- •For meeting analysis tool developers: Improve tools that track emotional trends in team meetings by providing second-level group emotion insights from audio and video.
Authors
Soma Iwata, Koji Inoue, Muyun Wu, Taiga Mori, Divesh Lala, Tatsuya Kawahara
Abstract
To realize natural behavior in dialogue agents in multi-party dialogue scenarios, it is important to understand group emotion such as valence and arousal as a whole. Most prior work addressed this task at the utterance level or using a coarse-grained time window, which is not sufficient to capture emotional dynamics. In this study, we formulate continuous recognition of the Group Emotion at a one-second resolution. Moreover, we also introduce the Mixed state, which captures the emotional divergence among participants in the group. We constructed a dataset with frame-level soft labels based on the TEIDAN corpus and propose a multimodal temporal framework that integrates audio and video information using a sliding-window context. Experimental results demonstrate that the temporal Transformer outperforms simple baselines and shows stronger temporal agreement with the ground-truth labels than the LLM-based model. The effect of context length is limited, whereas audio-visual input outperforms either unimodal input on the continuous-label metrics. Additionally, our analysis shows larger Group Emotion recognition errors in intervals with high Mixed values, exposing emotional divergence as a key challenge for group emotion recognition.