Benchmark tests emotion understanding in complex social interactions
AffToM: Benchmarking Recursive Affective Theory of Mind in Dynamic Multi-Party Interactions
Artificial Intelligence
Summary
People’s emotions in social situations depend on what they know and what they think others know or feel. The authors created a new test called AffToM to see how well computer models can follow and predict these complicated feelings as events happen and beliefs change. They also built a special system to help train these models better, especially when many people are involved in the interactions. Their tests show that understanding these layered emotions is still hard for machines, but their method improves performance noticeably.
What this means in practice
- •For interactive game developers: Integrate dynamic emotion tracking among multiple characters to create more realistic and emotionally responsive game narratives.
- •For customer service chatbot teams: Enhance chatbots’ understanding of customer emotions based on conversation history and inferred beliefs to improve responses in multi-user chats.
Authors
Jinpeng Hu, Yiduo Jin, Erqiang Wang, Jia Li, Jingjing Wu, Peipei Song
Abstract
Understanding emotion in realistic social interactions requires more than recognizing facial expressions or assigning affective labels. The same event can elicit different emotions depending on what each participant knows, what they believe others know or feel, and how these nested mental states evolve over time. Existing benchmarks largely study higher-order epistemic Theory of Mind (ToM) and multimodal emotion reasoning in isolation, leaving recursive affective reasoning under asymmetric and dynamically changing information underexplored. Therefore, we introduce AffToM, a benchmark for affective ToM with 1,922 test pairs and 19,759 training pairs. AffToM evaluates Individual Affective Reasoning (IAR), which infers emotions from events and socio-cognitive context, and Multi-party Affective Reasoning (MPAR), which recursively tracks participants' nested beliefs and emotions as new evidence emerges. We develop an event-driven multi-agent generation pipeline grounded in dynamic character interaction graphs that explicitly capture event sequences, interpersonal relations, and mental-state updates. Furthermore, we propose AffToM-R1, a two-stage post-training framework combining supervised fine-tuning with perspective-aware rubric-guided reinforcement learning. Experiments show that MPAR remains challenging, with even the strongest evaluated MLLM achieving less than 60% accuracy, while AffToM-R1 improves MPAR accuracy by 13.38 percentage points over its base model. Our dataset is available at: https://github.com/MindIntLab-HFUT/AffToM.