MemeTAG improves classification of harmful internet memes using keywords
MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction
Computer Vision and Pattern RecognitionMultimedia
Summary
Internet memes can sometimes spread harmful messages, and automatically detecting these is hard because memes combine pictures and words in complex ways. The authors created MemeTAG, a method that first finds descriptive keywords from memes using a model trained on pictures and language together. Then, they use a special network to combine these keywords into a summary that helps the system better connect the visual and text parts of memes. This method works better than previous ones on several meme datasets that focus on hate and harm.
What this means in practice
- •For social media platforms: Enhance automated detection of harmful memes by improving semantic understanding of images and text together.
- •For content moderation teams: Provide more accurate tools for classifying and filtering memes with hateful or harmful content.
Authors
Akshit Sharma, Prashant W. Patil
Abstract
The proliferation of harmful internet memes poses a significant societal threat, yet their automated classification remains a formidable algorithmic challenge due to the nuanced, multimodal nature of their content. To address this, we introduce MemeTAG, a novel dual-objective framework that pioneers a keyword-aware approach to meme classification. Our core innovation is a two-part semantic guidance mechanism: first, we leverage a pretrained Vision-Language Model to generate a set of descriptive keywords, that capture the high-level semantics. Second, we introduce the Aggregated Tag Inference Network (ATIN), an attention-based module that distills these keywords into a single, rich semantic embedding. This embedding serves as a target for a novel auxiliary reconstruction loss, which compels the model to learn deeply aligned visual and textual features. This approach, combined with an efficient three-stage training strategy, establishes a new state-of-the-art on the HarMeme, Hateful Memes Challenge (HMC), and PrideMM datasets, decisively outperforming existing state-of-the-art methods.