Multimodal search agents learn to handle longer and harder queries
MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents
Computer Vision and Pattern Recognition
Summary
Searching effectively with images and text together is a challenge, especially when questions are long and complex. The authors developed a method that makes training these search agents easier by transforming simple questions into harder ones and guiding the learning process carefully. Their approach uses reinforcement learning combined with a technique called on-policy distillation to improve how agents gather and reason over multiple types of information. This leads to better performance on complex, long-step multimodal searches compared to prior methods.
What this means in practice
- •For multimodal ai developers: Train AI agents to perform longer, more complex searches combining images and text with better reasoning through refined training data and joint learning.
- •For digital content search teams: Build more capable search tools that can handle detailed, stepwise queries requiring evidence from multiple media types like images and text.
Authors
Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
Abstract
Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.