AI summaryⓘ
The authors studied problems with vision-language models like CLIP when they face new types of images they weren't trained on, causing performance to drop. They found that commonly used test-time adaptation methods can actually make things worse over time. To fix this, the authors created a method called Local Margin Restoration (LMR) that carefully adjusts the model during testing to keep similar classes separated and avoid errors piling up. Their approach includes protections at both the single sample level and over the entire test data stream, resulting in better and more stable performance. They show through experiments that LMR works well even with small amounts of test data and outperforms other adaptation techniques.
Vision-Language ModelsCLIPTest-Time AdaptationDistribution ShiftZero-Shot LearningLocal Semantic GeometryMode CollapseAdaptive MarginBias Correction
Authors
Yan Huang, Guowei Wang, Xu Wang, Kangjun Liu, Xin Lin
Abstract
Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.