Multimodal models improve emotion understanding by reasoning through appraisals
From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding
MultimediaArtificial IntelligenceHuman-Computer Interaction
Summary
Understanding emotions just by looking at obvious facial or voice cues can be misleading, especially when those cues are hidden or mixed up. The paper’s authors created a new way for AI models to think about emotions more deeply by imagining how people evaluate events that cause feelings. They built a large dataset, a specialized model, and a new test to measure this understanding. Their approach helps computers better explain not only what emotion someone has but also why they feel that way. This makes emotion recognition more reliable and closer to how humans naturally think.
What this means in practice
- •For customer service chatbots: Improve chatbots' ability to interpret user emotions by reasoning about why users feel a certain way, not just by surface cues like text tone or emojis.$Commercial implications: It enables chatbots to offer more empathetic and context-aware responses, enhancing customer satisfaction and retention.
- •For mental health app developers: Develop apps that better assess users' emotional states by combining multimodal signals with reasoning about underlying causes of emotions.
Authors
Jia Li, Yichao He, Yangchen Yu, Qiankun Li, Xinyi Li, Baiyi Ye, Zhenzhen Hu, Richang Hong, Erik Cambria
Abstract
Recent multimodal large language models (MLLMs) increasingly incorporate explainable reasoning for emotion understanding. However, reasoning based mainly on observable affective cues can reduce emotion understanding to superficial cue-label associations, giving rise to the Clever Hans effect. Such shortcuts become unreliable when affective cues are implicit, conflicting across modalities, linguistically misleading, or obscured by redundant details. In contrast, human emotions are shaped by how individuals interpret and evaluate surrounding events beyond observable cues. Inspired by appraisal theories of emotion, we formulate multimodal emotion understanding as a progression from perception to cognitive appraisal, and introduce a dataset, a model, and a benchmark to support this novel paradigm. CogEmo-40K is a large-scale instruction-tuning dataset constructed through a perception-to-appraisal pipeline to elicit evidence-grounded reasoning across six cognitive appraisal dimensions underlying emotion. CogEmo-MoE is a compact sparse MLLM that introduces interleaved MoE blocks for appraisal-specific adaptation, enabling effective appraisal reasoning at a substantially smaller scale than typical emotion MLLMs. CogEmo-Bench introduces an Appraisal Evidence Quality Score (AEQS) to assess cognitive-affective understanding across six complementary appraisal dimensions, addressing the limitation of conventional emotion metrics that evaluate what emotion is predicted but not why it arises. Extensive experiments show that our paradigm not only leads CogEmo-Bench, but also exhibits strong cross-domain generalization. Our findings suggest that perception-to-appraisal reasoning can move beyond surface-level cue-label associations toward more reliable multimodal emotion understanding and closer cognitive alignment between MLLMs and humans.