Knowledge-driven system improves detection of harmful memes online
MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection
Artificial Intelligence
Summary
Harmful memes on social media often use tricky combinations of pictures, words, and background info, making them hard to spot. The authors created MemeLogos, a system that builds a knowledge graph from lots of harmful meme examples to identify common harmful patterns and reasoning steps. This system uses that knowledge like a guide to reason better about new memes, making detection more accurate and easier to explain. It outperforms older methods by combining structured knowledge with a step-by-step reasoning process inspired by human thinking.
What this means in practice
- •For social media platform teams: Integrate structured harmfulness knowledge for more reliable and interpretable meme moderation tools.
- •For content moderation service providers: Enhance moderation systems by guiding meme harmfulness decisions through explicit reasoning patterns extracted from varied harmful content.$Commercial implications: Enables building commercial moderation solutions with improved accuracy and explainability for detecting harmful memes.
Authors
Hexiang Gu, Saihui Hou, Changlan Liu, Min Ren, Huijia Wu, Hongxun Gu, Zhaofeng He
Abstract
With the rapid proliferation of social media, harmful memes have become a growing threat to online ecosystems. Their harmfulness is often implicitly conveyed through complex interactions among visual, textual, and contextual cues, making accurate detection challenging. Recent studies have introduced Retrieval-Augmented Generation (RAG) and external knowledge into harmful meme detection. However, existing methods mainly retrieve similar instances or auxiliary information, limiting their ability to capture reusable harmfulness knowledge and underlying reasoning mechanisms, and thus constraining both detection performance and interpretability. To address these challenges, we propose MemeLogos, a knowledge-driven retrieval-augmented reasoning framework for harmful meme detection. MemeLogos abstracts recurring harmfulness patterns and reasoning structures from diverse harmful cases into reusable knowledge units and organizes them into a dedicated star-shaped knowledge graph. By combining structured knowledge retrieval with an explicit multi-stage reasoning process inspired by cognitive mechanisms, MemeLogos leverages retrieved knowledge to guide harmfulness reasoning, improving both detection reliability and interpretability. Extensive experiments show that MemeLogos consistently outperforms existing harmful meme detection methods while generating more coherent and interpretable reasoning, demonstrating the effectiveness of structured harmfulness knowledge for multimodal content understanding.