Visual routing method greatly improves multi-step image retrieval for agents

Learning to Route in Visual Space via Multi-Step Embedding Retrieval

Artificial IntelligenceInformation Retrieval

Summary

Finding information in pictures step-by-step is hard when agents have to ask questions at every stage. The authors show that letting a retrieval system navigate images in multiple steps directly, without needing new text queries each time, works much better. They built a benchmark called VHOP and a new system, VHOP-Router, that retrieves image chains in one call. This approach improved success rates a lot, used much less text, and kept working well on new problems.

What this means in practice

  • For computer vision engineers: Improve multi-step visual search tools by embedding retrieval that reduces reliance on text queries and boosts accuracy in finding image sequences.
  • For ai assistant developers: Enhance AI assistants' ability to navigate visual content over multiple steps with less data input and higher success rates.

Authors

Tianyu Chen, Mingyuan Zhou, Jiaxing Wu

Abstract

LLM agents rely on retrieval tools to access external knowledge, yet visual agentic search remains severely bottlenecked by standard single-step retrievers. In current pipelines, the agent must issue text queries for every intermediate step, struggling when visual clues are difficult to describe or when the retriever fails to surface necessary intermediate evidence within its top results. We hypothesize that offloading multi-step navigation across the entire embedding space directly to the retrieval tool resolves this performance bottleneck. To study this systematically, we introduce VHOP, a flexible data generation framework and benchmark with five core difficulty levels testing both visual matching and search planning. Using this framework, we develop VHOP-Router, an end-to-end training pipeline---combining supervised fine-tuning, online imitation learning, and reinforcement learning---that transforms a standard embedding model into an autoregressive multi-step retriever. Operating directly in the visual latent space, VHOP-Router retrieves linked image chains in a single tool call without requiring the agent to formulate intermediate text queries. Experiments show VHOP-Router boosts retrieval performance from under 5\% to 76.3\%. In agentic search, it improves task success rates by 52.7\% and reduces the average token length by 61\% from 1886 to 728, whereas upgrading the agent yields only a 3.7\% gain. Compared to a strong baseline where the agent retrieves the top 50 results per step, VHOP-Router maintains superior performance while reducing in-context images by $23\times$ and cutting the cumulative API payload by $35\times$. The models also generalize robustly to unseen difficulty levels and realistic test sets. Ultimately, VHOP and VHOP-Router provide an efficient and effective solution for visual agentic search that leaves native LLM capabilities entirely intact.