Selective adaptation strategy improves vision language model accuracy
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Vision-language AI models sometimes struggle when shown new types of images or captions. The authors found that adapting these models to each test example can often do nothing or even hurt accuracy. They suggest a new way to decide which examples should be adapted by checking how much the model’s answers change when images are altered slightly. Adapting only when changes are big keeps or improves accuracy and reduces unnecessary work.
What this means in practice
- •For machine learning engineers: Reduce computation and improve model predictions by selectively adapting vision-language models only on impactful test samples during deployment.
- •For mobile app developers: Implement light-weight vision-language features that adapt less frequently to save power and latency without losing accuracy.
Authors
Siru Jiang, Yuwei Liang, Jian Liang, Ran He, Tieniu Tan
Abstract
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.