Multimodal models get better at avoiding false emotion guesses
Multi-Faceted Evaluation and Mitigation of Emotion Hallucinations in MLLMs
Multimedia
Summary
Multimodal large language models can understand emotions but sometimes mistakenly invent feelings that aren’t really there, called emotion hallucinations. The authors created a new way to measure these mistakes across different types of emotional reasoning. They found that many fixes only address some mistakes but make others worse. To improve this, they designed a new method that remembers and targets specific errors during emotion understanding to reduce hallucinations more precisely. Their experiments show this method works on many types of models.
What this means in practice
- •For social media platform engineers: Reduce incorrect emotional interpretations in multimedia content moderation tools to improve accuracy of automated emotion detection.
- •For interactive chatbot developers: Enhance chatbots to provide more accurate emotional responses by minimizing false emotion attributions in user interactions.
Authors
Bowen Zeng, Peipei Song, Weidong Chen, Shengeng Tang, Song Ye, Yuanhong Zhong, Beier Zhu, Xun Yang
Abstract
Multimodal large language models (MLLMs) have shown strong potential in open-ended emotion understanding, yet they often generate emotion hallucinations. Evaluating such hallucinations is particularly challenging for two reasons. First, emotion understanding spans multiple cognitive facets, from multimodal perception to psychological reasoning. Second, emotional interpretations are expressed in free-form language, making existing closed-ended protocols insufficient for evaluation. To address these challenges, we introduce EHR (Emotion Hallucination Rate), an evaluator that quantifies emotion hallucinations across six facets: expression, action, audio, instinct, logic, and conclusion. Using EHR, we reveal that existing mitigation methods often reduce hallucinations in some facets while aggravating them in others, exposing the limitation of coarse-grained correction and the need for facet-aware localization and mitigation. Motivated by this finding, we propose HMER (Hallucination-aware Memory-guided Emotion Reasoning), a training-free framework for emotion hallucination mitigation. HMER maintains a Hallucination Memory that records localized hallucinated claims and enables targeted logit rectification, together with an Anchor Memory that preserves reliable intermediate reasoning states to stabilize subsequent generation. By selectively suppressing unreliable cues while preserving trustworthy reasoning context, HMER enables fine-grained mitigation across diverse hallucination facets. Extensive experiments on 19 MLLMs demonstrate the prevalence of emotion hallucinations and the effectiveness of our framework across diverse model architectures.