Multimodal agents choose reasoning steps to speed up search tasks
Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents
Machine LearningArtificial Intelligence
Summary
When computers try to solve problems that involve text and images, they usually explain their reasoning in long sentences before acting, which can be slow and not always helpful. The authors propose a way for these systems to pick from a list of usual reasoning steps instead of writing them out each time. This method makes the process much faster while still getting good results in finding information or objects. They tested this on several benchmarks and showed it works well with less delay.
What this means in practice
- •For multimodal ai developers: Implement efficient reasoning selection to speed up multimodal search agents without losing accuracy.
- •For software engineers in ai products: Reduce the latency of AI-powered search tools that handle text and images by adopting selection-based reasoning.
Authors
Feiyu Gavin Zhu, Xiaoyu Zhu, Jiqi Yang, Rui Yang, Arnab Kumar Mondal, Yancheng Wang, Xinke Deng, Jean Oh, Reid Simmons, Joerg Liebelt, Xiang Kong, Zhongyu Jiang
Abstract
Multimodal agents commonly generate free-form reasoning before each action. For small models, limited model capacity can result in lengthy reasoning that provides little useful guidance for action generation while incurring substantial inference cost. To address this challenge, we introduce Selection-based Structured Reasoning (SSR), a framework that reformulates reasoning as selection instead of open-ended generation. SSR represents recurring high-level reasoning as pre-specified, reusable natural-language candidates. At each turn, the model selects from these reasoning candidates based on their likelihoods given the current context, without requiring an auxiliary task head. Using pre-specified reasoning traces enables parallel scoring, where teacher-forced prefilling computes token likelihoods concurrently within and across candidates using a shared context KV cache. We evaluate SSR on seven multimodal search benchmarks using 2B and 4B models. Across multiple reinforcement learning objectives and supervised fine-tuning, SSR delivers significant efficiency gains without sacrificing task performance. SSR achieves an average success rate competitive with leading search agents of the same scale, while reducing per-turn reasoning latency by over 90% and total per-question model inference latency by 28-54%. Project page: https://zfy0314.github.io/ssr-webpage/.