An Effective Router for Vision-Language Model Selection
2026-06-08 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address the problem of choosing the best vision-language model (VLM) from many options, which can be confusing due to varying performance and needs. They create a new dataset showing how seven popular VLMs handle over 30,000 image-text tasks. They then design ARMS, a system that helps pick the right VLM by better understanding the inputs and the models' strengths. ARMS can also adapt to new models easily with special training methods, showing strong results even against much larger commercial models like GPT-4o. Their work helps make VLM selection more practical and efficient.
Vision-Language ModelsModel SelectionRouterMultimodal DatasetFeature RepresentationIncremental TrainingOut-of-DistributionGPT-4oModel AdaptationPerformance Paradox
Authors
Can Wang, Shengwei Wang, Bolin Zhang, Zhiying Tu, Dianhui Chu
Abstract
Vision-language models (VLMs) with varying performance and resource requirements are widely deployed, making it difficult for users to select the most appropriate one among numerous VLM candidates. Existing work reveals the performance paradox phenomenon in language models and focuses on routing methods to solve it. However, developing a router for VLM selection is still a critical yet challenging problem, which primarily faces: 1) lack of specialized data, 2) ineffective feature representation, and 3) rigid model space and costly adaptation. In this paper, we construct a multimodal dataset for VLM selection, containing the outputs of seven mainstream VLMs on 32,626 unique image-text queries. We then propose ARMS, a router for VLM selection. ARMS enhances input signals with VLM profiles, employs a simple but effective architecture to improve representations of queries and VLM capabilities. To improve ARMS' adaptation to new VLMs, we propose two extension training strategies: incremental training and independent training. Experimental results on both in-distribution and out-of-distribution test sets demonstrate the effectiveness of ARMS. In particular, using our training strategy, ARMs (only 800M in size) can adapt to a broader VLM space and defeat commercial models like GPT-4o that are hundreds of times larger in scale. Our code, models, and datasets are available in the anonymous repository.