Multimodal reasoning agents learn to verify evidence and improve tool use
Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Sometimes, computer programs that answer complex questions use tools like web search but struggle when they need to look at both text and images and check their own facts. The authors developed a new training method called SVRL that helps these programs learn to double-check the information they find and decide when to use tools more wisely. This approach makes the programs better at answering tricky questions involving multiple steps while using fewer resources. Their method improves performance even with a small amount of training data and narrows the gap between smaller models and larger ones.
reinforcement learningmultimodal agentsevidence verificationtool usevisual question answeringsearch queriesreinforcement learning finetuningmulti-hop reasoning
Authors
Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray
Abstract
Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.