Autoregressive retriever boosts multimodal search by refining queries iteratively
Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval
Information RetrievalComputer Vision and Pattern Recognition
Summary
Multimodal search systems usually find items by matching a user’s query to stored information only once, which can miss important details. The authors developed a method called AutoRegressive Retriever (ARR) that improves search by successively picking helpful items and updating the query to better understand what the user wants. This process continues through several steps before producing a final search result, making the search more accurate. They train ARR using special techniques so it learns how to use feedback effectively.
What this means in practice
- •For search engine developers: Improve search accuracy by updating queries with feedback from retrieved items in multimodal information retrieval systems.
- •For recommendation system engineers: Enhance recommendation quality by iteratively refining user queries based on intermediate item feedback to better capture user preferences.
Authors
Jianfei Zhao, Yifan Wang, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan, Yang Luo, Boyuan Pan, Xu Kai, Yao Hu
Abstract
Universal multimodal retrieval typically encodes a query once and ranks independently indexed items by embedding similarity. This design supports efficient search, but leaves the query representation unchanged even when retrieved items could help clarify the information need. We introduce the AutoRegressive Retriever (ARR), a multimodal retrieval model that learns both to select informative items and to use their content to refine subsequent retrieval. ARR alternates between retrieving an item and updating the query embedding, then uses the final embedding to rank the collection. Supervised fine-tuning teaches the encoder to use feedback through stepwise contrastive supervision. Reinforcement learning treats feedback items as actions and optimizes their selection using the final reciprocal rank of a relevant item. A query-side adapter enables this optimization against a fixed item index. ARR demonstrates strong retrieval performance on both in-domain and zero-shot benchmarks, outperforming the compared baselines on average. Further analyses show that feedback improves retrieval at inference time and that training with feedback also improves the initial query embedding, before any item is observed.