Multilevel analysis improves fake news detection across image text conflicts
Decoupling Consistency and Conflict across MLLM Layers for Fake News Detection
Computer Vision and Pattern RecognitionComputation and Language
Summary
Fake news often has mismatches between images and text that don’t line up, like wrong objects or claims. The authors propose a new way to spot these conflicts by looking at different layers inside a big language model used for both images and text. They pick out important parts and track how understanding changes through the layers, separating signs of truth and conflict. This lets their method better detect fake news than older approaches.
What this means in practice
- •For social media content teams: Enhance automated tools to flag misleading posts by detecting subtle mismatches between images and text in news stories.
- •For content moderation platforms: Improve detection systems that evaluate the trustworthiness of multimodal posts by using layer-wise conflict analysis.
Authors
Kai Qian, Dongjie Xu, Weijie Shi, Jiaqi Wang, Haolun Ma, Jia Zhu, Jiajie Xu
Abstract
The trustworthiness of Web content faces a growing threat from multimodal fake news. Real news usually shows aligned image-text semantics, while fake news often contains a few conflicts at varying levels, such as mismatches in objects, scenes, events, and claims. However, existing methods usually treat consistency and conflict in a coarse and static way, limiting their ability to capture sparse conflicts and adapt to varying conflict levels. Inspired by the distinct knowledge encoded across layers in large language models, we propose Multi-Level Consistency and Conflict Decoupling (MLCCD), which captures conflicts at different levels from different layers of multimodal large language models (MLLMs). Specifically, we first use the MLLM's final decision representation to select visual and textual tokens relevant to the task, preventing irrelevant content from diluting sparse conflicts. Then, we preserve tokens whose representations change sharply between adjacent layers, capturing the knowledge each layer adds. Finally, we decouple this knowledge into consistency and conflict evidence and aggregate each across layers with its own router, adapting to varying conflict levels. Experiments on three benchmarks show that MLCCD outperforms existing methods.