Vision language models improve image recognition across different settings
Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models like CLIP can recognize images without extra training, but their accuracy drops when images look different than those seen originally. The authors propose a method called Domain Recentering with Confidence Calibration (DRC) that adjusts the model using unlabeled new images. DRC improves accuracy by repositioning image features and correcting biases without needing additional labels. This method works better than previous approaches, especially when images come from diverse or shifted sources.
What this means in practice
- •For computer vision engineers: Increase robustness of image classifiers to new, unlabeled datasets without retraining by applying DRC calibration to vision-language model embeddings.
- •For autonomous vehicle teams: Improve real-world object recognition under varying visual conditions by using DRC to adapt vision-language models without additional annotated data.
Authors
Youngeun Seol, Jimin Shin, Heeseo Yoon, Uiwon Hwang
Abstract
Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts.