Multimodal AI improves fine emotion recognition with visual pruning
Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning
Computer Vision and Pattern Recognition
Summary
Understanding subtle emotions from images is hard for current AI systems because they usually look at the whole picture and miss small clues. The authors created a method called Decoding Affective Nuances (DAN) that helps AI better focus on important visual details and tell apart very similar emotions by comparing key image parts. DAN works without extra training and boosts emotion recognition, especially in cases with many fine emotions to choose from. This advancement helps AI models get closer to human-level emotional understanding.
What this means in practice
- •For social media companies: Improve automatic detection and categorization of subtle emotions in user-generated images to enhance content moderation or user experience personalization.
- •For human-computer interaction designers: Enhance emotion-aware interfaces by enabling systems to better distinguish nuanced emotional expressions from visual inputs during real-time interactions.
Authors
Cheng Ye, Weidong Chen, Zhaobo Qi, Beier Zhu, Zhendong Mao
Abstract
While multimodal large language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual Pruning (CDVP), which isolates discriminative visual tokens to reason the final emotion category by computing the absolute discrepancy between the attention distributions of similar emotions. Performances on several benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25 dataset that contains 25 fine-grained emotion categories.