Trace framework tracks emotions from causes to effects in social scenes
Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes
Artificial Intelligence
Summary
Understanding emotions is more than just recognizing if someone is happy or sad; emotions develop through causes, feelings, and effects involving thoughts and social cues. The authors introduce TRACE, a new way to analyze emotions by breaking them into stages: the situation, the emotional response, and the consequences. They test current AI models with a large video-based benchmark and find these models struggle with reasoning about emotions fully. To improve this, they propose TRACER, a method that explains its reasoning step-by-step, outperforming other models on various tasks about emotions in social settings.
What this means in practice
- •For multimodal ai developers: Improve AI systems to better understand emotional causes and effects in social videos by integrating cognitive reasoning steps.
- •For human-computer interaction teams: Enhance empathetic responses in social robots or avatars by tracing emotion development stages from context to reaction.
Authors
Hao Li, Jinye Zhang, Bobo Li, Mong-Li Lee, Wynne Hsu, Zheng Wang, Hao Fei, Min Zhang
Abstract
Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE, a cognition-oriented framework that formalizes an affective episode through three interrelated stages: Condition, Affect, and Effect, integrating observable cues with cognitive factors such as internal stance and regulation of emotional display. Based on this formulation, TRACE-Bench evaluates multimodal models in real-world social scenes through five tasks spanning grounded affect recognition, regulation decoding, cause reasoning, effect reasoning, and full-chain reconstruction, with 3,746 structured question-answer pairs over 646 videos. A matched human-model comparison reveals a substantial performance gap, while affect-specialized models also generally lag behind general-purpose MLLMs. Model outputs show recurring failures, including treating displayed behavior as genuine feeling and fabricating unsupported events during long-chain generation. We further propose TRACER, a cognition-grounded structured reasoning method that couples each inference with explicit premises from factual observations, cognitive appraisals, and established upstream conclusions, forming a traceable graph of intermediate and target conclusions. TRACER outperforms all evaluated model baselines on each of the five tasks. Project page: https://cogaffc.github.io/TRACE