Multimodal misinformation detection improves with tool guided learning
MM-VeriAgent: Learning to Use Extensive Tools to Verify Multimodal Misinformation with Reinforcement Learning
Computer Vision and Pattern Recognition
Summary
Detecting false information that mixes text and images is tricky because it can come from multiple kinds of fakes. The authors created a toolkit of specialized models for checking text, images, and their combinations. They then trained an AI agent to learn how to use these tools together efficiently, improving its ability to spot mixed-source misinformation. A smart caching system also sped up training by reusing previous results instead of running all checks repeatedly.
What this means in practice
- •For social media platforms: Improve detection of complex fake news combining text and images by automatically selecting the best analysis tools.$Commercial implications: Enables advanced misinformation detection products for social networks to reduce fake content spread.
- •For fact-checking teams: Provide a more adaptable system that learns to apply multiple verification tools selectively to mixed-media claims.
Authors
Peipei Li, Shuhan Xia, Shengyang Liu, Zekun Li, Ran He
Abstract
Real-world multimodal misinformation often involves mixed forgery sources, requiring sample-specific detection strategies. Existing tool-augmented methods rely on predefined workflows or inference-time planning, limiting adaptability or increasing inference cost. To address this issue, we introduce \textbf{MM-VeriAgent}, which learns to verify mixed-source multimodal misinformation with tools. We first build \textbf{MM-VeriTools}, a specialized toolkit for misinformation detection agents. By benchmarking various candidate models and methods on the sub-tasks required by mixed-source detection, we select the strongest for textual, visual, and cross-modal forgery analysis and encapsulate them as callable tools with a unified interface. On top of this toolkit, we train the LVLM agent with reinforcement learning to teach it how to use these tools to better solve mixed-source detection. Since many of the tools are specialized models whose online execution at every rollout severely limits RL efficiency, we further introduce \textbf{Tool-Execution Cache}, which pre-executes candidate tool calls and reuses their cached outputs during training. This preserves multi-step rollouts while reducing online tool execution, largely improving the training efficiency.Experiments on MMFakeBench demonstrate substantial accuracy gains over the base model without explicit tool search at inference time. Ablation and efficiency analyses further validate the learned tool-use policy and show that Tool-Execution Cache reduces online tool executions during training.