Event context improves emotion recognition beyond visual cues
AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances
Computer Vision and Pattern Recognition
Summary
Understanding emotions just by looking at someone's face or reaction can be tricky because the same expression might mean different feelings depending on what happened. The authors created a way to recognize emotions by also figuring out the event that caused them, not just relying on visual clues. They found that knowing the event almost doubled human accuracy in guessing emotions. Their method, AffectReveal, checks multiple pieces of evidence about the event and combines that with the original image or video to better predict emotions without extra training.
What this means in practice
- •For social media platforms: Automatically detect nuanced user emotions in videos for better content moderation by interpreting event context beyond facial expressions.$Commercial implications: Enables emotion-aware moderation tools that understand context, improving platform safety and user experience.
- •For human-computer interaction teams: Enhance emotion-recognition systems in interactive agents by integrating event context, not just visual appearance, to respond more appropriately.
Authors
Yihao Qian, Runhao Zeng, Sicheng Zhao, Feng Liang, Hongmin Cai, Mingkui Tan
Abstract
Visual emotion recognition commonly assumes that all evidence required for prediction is contained in the observed image or video. Yet the same visible reaction can convey different emotions depending on events beyond the input: tears, for example, may indicate grief or joy. We formulate Event-Grounded Emotion Recognition (EGER), where emotion recognition requires recovering the affect-determining event. We construct EGER-Bench, comprising 10,052 videos and 10,734 images across 11 emotions, two source domains, and four visual settings. A study with six annotators shows that event context raises human recognition accuracy from 33.96% to 72.08%, confirming that visual evidence alone is often insufficient. Semantic relevance alone does not solve EGER: a plausible event may imply the wrong emotion if its identity, focal-person role, relationship, or outcome is misinterpreted. We therefore propose AffectReveal, a tuning-free framework that first constructs and independently verifies evidence-grounded alternatives over these affect-critical factors. It then cross-checks the recovered event against face-masked in-media facts through bidirectional atomic evidence support, while retaining the original unmasked input for final prediction. Across three downstream models and four input settings, AffectReveal yields average UAR gains of 5.26--10.53 points. For three fine-tunable models, it also enables untuned models to outperform their fine-tuned visual-only counterparts in all 12 accuracy comparisons, without updating downstream parameters.