Social media emotions expressed better with event context and images
Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts
Computation and Language
Summary
People often share their feelings on social media using both pictures and words. This paper looks at how well readers can guess the author’s true emotions by reading these posts. The authors show that knowing the real event behind the post helps people and computers understand the emotions more accurately. However, it is still hard to figure out the feeling, especially when the pictures carry much of the emotional message.
What this means in practice
- •For social media platform teams: Improve emotion recognition in posts by combining image understanding with event context to better detect user feelings.
- •For advertising teams: Design targeted campaigns by accurately interpreting the emotional tone of multimodal user posts that reference real-life events.
Authors
Christopher Bagdon, Carina Silberer, Roman Klinger
Abstract
Emotions are an essential aspect of human communication, particularly on social media, where authors frequently combine text and images to convey their emotions. Yet prior work on emotion analysis of social media posts has overlooked two important aspects in regard to measuring how well readers can reconstruct the authors' intent: (1)~the image modality, with most work focusing solely on text, and (2)~the real-world events that trigger the expressed emotions, and their relationship to the post content. We therefore study the relation between (a) the author's experience of the event that caused them to write a social media post and (b) the content of the post, with a focus on readers' capability to reconstruct that emotion expression. To do that, we introduce the Multimodal Multi-Emotion-Model dataset Mult2EMo, created by collecting annotations from both authors and readers on the posts and their triggering events. We find that reconstruction is possible but challenging for both human readers and computational models. We show that understanding the triggering event is crucial for accurate reconstruction, and that reconstruction is particularly challenging when posts rely heavily on the image to express emotion.