Multimodal agents learn to verify and improve search using reinforcement learning
Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Multimodal reasoning agents that use tools like web search struggle to use noisy information from images and text correctly. The authors developed a method called SVRL that helps these agents teach themselves to check and filter the information they find during their reasoning process. This learning method also encourages the agents to only make useful searches and to ask better questions to improve their answers. Testing on a visual question answering task showed that SVRL helps the agents become more accurate and efficient, closing the gap with much larger systems while using less computing power.
What this means in practice
- •For ai system developers: Develop smaller multimodal agents that independently verify and filter search results to improve reasoning accuracy and efficiency.
- •For chatbot developers: Build chatbots that generate more diverse and relevant queries to external tools for better information retrieval in complex tasks.
Authors
Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray
Abstract
Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.