Summary
Emotional Video Captioning tries to describe videos both accurately and with the right feelings. Existing methods sometimes get confused by backgrounds, mistakenly attaching wrong emotions to scenes due to repeated patterns like sunny beaches linked to happiness. The authors created a new dataset with detailed labels pinpointing true emotional causes in videos and designed a method that separates these real causes from distracting backgrounds. Their approach uses a special module that focuses on motion and contrasts real versus fake scenarios to make emotion predictions more trustworthy. This helps captions reflect genuine feelings tied to what actually happens in the video, not just the scenery.
What this means in practice
- •For video content platforms: Improve video description systems to generate emotionally accurate captions by focusing on genuine causes rather than misleading backgrounds.
- •For human-robot interaction teams: Enhance robots’ understanding of human emotions in videos by grounding emotional cues in real visual causes instead of context biases.
Authors
Cheng Ye, Weidong Chen, Peipei Song, Zhendong Mao
Abstract
Emotional Video Captioning aims to generate factually accurate and emotionally empathetic descriptions. While recent methods have recognized the importance of visual causes to guide emotion perception and caption generation, they fundamentally rely on simple attention matching, which inevitably suffers from {causal redundancy and spurious correlations} in co-occurrence bias (e.g., misclassifying ``sadness'' as ``joy'' on a sunny beach), leading to severe shortcut learning from confusing backgrounds. Furthermore, existing evaluations fail to verify whether models have genuinely mastered causal reasoning or merely exploited background confounders. To address these limitations, we first construct {EVC-CauseGround}, a comprehensive benchmark with dense spatio-temporal causal annotations. Crucially, it introduces a carefully selected {Causal-Faithfulness Subset} to explicitly quantify genuine emotion-cause attribution. Second, we propose {Causal-EVC}, an emotion-grounding captioning framework, which introduces a Motion-guided Causal Spatiotemporal Localization module to precisely decouple causal triggers from background confounders. Besides, we introduce an Interpretable Sparse Emotion Routing module. By synthesizing counterfactual representations and formulating a novel counterfactual contrastive objective, we enforce the model to anchor its emotion predictions strictly on authentic causal triggers instead of confusing background. Extensive experiments show that Causal-EVC not only achieves the best performance on semantic metrics but also exhibits significant advantages in the causal-faithfulness subset, which demonstrates that our model could mine emotional cues from genuine visual causes and mitigate co-occurrence bias for interpretable multimodal emotion understanding.