Language based method improves vision language model selection accuracy
Good Rankings, Poor Choices? Rethinking Language-Only Vision-Language Model Selection
Computer Vision and Pattern RecognitionMachine Learning
Summary
Choosing the best vision-language model for a task can be tricky when only text information is available. The authors found that common ways to rank these models don’t really show how good the chosen model will be. They suggest a new way to measure how much worse the chosen model is compared to the best one, and a new method to pick models based on this measure. Testing on many tasks and models, their approach picks better models more often than previous methods.
What this means in practice
- •For machine learning engineers: Select vision-language models more reliably using text-only methods when adapting models to new tasks without testing data.
- •For automated model deployment teams: Improve automated pipelines that choose models for computer vision tasks by reducing performance selection errors using the new regret-based metric.
Authors
Hyuk Kwon, Jaebyeong Jeon, Hyeonseo Jang, Kibok Lee
Abstract
Language-Only Vision-Language Model Selection (LOVM) aims to identify the Vision-Language Model (VLM) with the highest zero-shot accuracy on a target task using only textual information. We revisit the existing LOVM evaluation protocol and argue that ranking metrics, such as top-5 recall and Kendall's rank correlation, do not directly assess the performance of the selected model. We therefore adopt simple regret (SR), defined as the performance gap between the best and selected models, and introduce Outlier-Robust Normalized Regret (ORNR) to account for the varying performance ranges across tasks while reducing sensitivity to outliers. Building on this revised evaluation protocol, we propose Target-Conditioned Expected Source Regret (TC-ESR), a training-free method that selects the model with the lowest similarity-weighted mean regret across source tasks. Across 22 target tasks and 43 VLMs, TC-ESR achieves the lowest mean SR and ORNR among all compared methods. Separately, we identify two sources of target performance leakage in the prior ranking-based evaluation protocol and provide corrections that remove both.