MemePIVOT improves detection of harmful cues in multimodal memes
Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification
MultimediaComputation and LanguageMachine Learning
Summary
Memes often rely on small details in their images or text to carry harmful, hateful, or sarcastic messages. The authors created a new dataset that labels these important details to help computers learn what to look for. They also designed a new method, MemePIVOT, which focuses on these key parts by linking specific words to parts of the image while also considering the overall meme context. Their approach performs better than previous methods in detecting harmful content in memes. This shows that paying attention to small but crucial clues helps computers understand memes more accurately.
What this means in practice
- •For social media platforms: Automatically detect harmful or hateful memes by focusing on key visual and textual clues for better content moderation.$Commercial implications: Enables development of improved moderation tools that reduce platform liability and maintain healthier online communities.
- •For digital advertising teams: Identify sarcastic or harmful meme content before ad placement to avoid brand reputation risks and unwanted associations.
Authors
Akshit Sharma, Prashant W. Patil
Abstract
Memes often derive their harmful, hateful, or sarcastic meaning from small but decisive visual, textual, or cross-modal cues. Existing multimodal classifiers can miss such evidence when relying mainly on global image-text representations. We introduce MemeCF, a cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence. We also propose MemePIVOT, a local-global architecture for meme classification. MemePIVOT uses frozen CLIP features, unbalanced optimal transport to align words with image patches while allowing irrelevant evidence to remain unmatched, and an evidential fusion head to combine local grounding with global meme context under uncertainty. Experiments on HarMeme, PrideMM, and MemeCF show consistent gains over strong text-only, image-only, multimodal, and vision-language baselines. Cross-dataset and ablation results further show that explicit pivotal-evidence modeling improves robustness and contributes meaningfully beyond global multimodal representations. Our code and dataset are publicly available at https://github.com/AkshitSharma1/MemePIVOT