Multimodal large language models detect hidden emotions using context
Beyond Multimodal Emotion Conflict: Temporal Context-Aware Hidden Emotion Detection with Multimodal Large Language Models
Artificial Intelligence
Summary
People show emotions in different ways through words, voice, and facial expressions, but sometimes these signals don’t match. This mismatch can happen for many reasons, not just because someone is hiding how they feel. The authors want to tell the difference between regular mixed emotions and when someone is actually concealing their feelings. They plan to create a special dataset labeled with true and hidden emotions, and build a model that uses facial details, multiple signals, and time context to find hidden emotions better.
What this means in practice
- •For mental health professionals: Improve therapy tools by identifying patients’ concealed emotions through speech, face, and text signals analyzed over time.
- •For human resource teams: Enhance employee feedback systems by distinguishing genuine emotions from concealed feelings using multimodal cues and context awareness.
A position paper. It proposes an approach and reports no results.
Authors
Yonghao Wang, Jiaxin Zhou
Abstract
Multimodal emotion recognition commonly combines textual, acoustic, and visual signals to infer affective states. Recent work has begun to study conflicts among modalities, but a conflict does not necessarily indicate that a person is concealing an emotion. It may also result from mixed emotions, individual expressive styles, or contextual factors. This proposal aims to distinguish ordinary emotional inconsistency from emotion concealment and to infer the underlying emotion when supported by evidence. We plan to construct a multimodal benchmark with explicit annotations of expressed emotion, internal emotion, and concealment conditions, and to develop an MLLM-based framework that integrates fine-grained facial Action Units (AUs), cross-modal features, and temporal context.