Vision language reranking gets faster with adaptive computing budget
AMBER: Multi-View Adaptive Budget Allocation for Listwise Vision-Language Reranking
Artificial Intelligence
Summary
Vision-language models help improve search results by reordering images or text based on their relevance. But using these models takes a lot of computing power, which limits how many results they can look at closely. The authors propose a method called AMBER that smartly decides where to spend computing resources, focusing more on uncertain or important comparisons. This method uses an evolving ranking system inspired by game scoring and decides which queries and views to process to get the most information while saving time. Tests show it achieves better accuracy with the same or less computing effort compared to other methods.
What this means in practice
- •For image search engineers: Optimize resource use to improve image search result rankings with limited computing capacity.
- •For mobile app developers: Enhance multimodal search features on mobile devices by efficiently allocating computation for better ranking insights.
Authors
Wenteng Chen, Jiachen Zhu, Rong Shan, Tianyi Xu, Yuxiang Chen, Congmin Zheng, Teng Wang, Junjie Wu, Weiwen Liu, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin
Abstract
Vision-language models (VLMs) are powerful listwise rerankers for multimodal retrieval, but high inference costs restrict them to evaluating small local candidate views. Existing multi-call strategies rely on fixed schedules, wasting expensive VLM calls on uninformative candidate pairs and easy queries. To address this, we propose Adaptive Multi-view Budgeted Elo Reranking (AMBER), an online, budgeted multi-view reranking framework that dynamically optimizes global resource allocation. AMBER treats fragmented listwise VLM outputs as local tournaments, using continuous Elo updates to maintain a lightweight global ranking state. Building on this, it allocates computation at two levels: dynamically constructing candidate views with high score ambiguity, and scheduling queries to maximize expected information gain. We show that each Elo update corresponds to a stochastic gradient ascent step on the Bradley-Terry log-likelihood, and provide a submodular information-theoretic motivation for the query-level allocation strategy. Experiments on CIRR, CIRCO, and PhotoBench demonstrate that AMBER achieves the strongest overall performance among the compared multi-call VLM reranking methods under comparable VLM-call budgets, while remaining effective in lower-budget settings. Our code is publicly available at https://github.com/wnlfc/AMBER.