Multimodal anchors improve test-time adaptation under distribution shifts
Sparse MLLM Anchors, Dense Adaptation: Breaking the Self-Referential Loop in Wild Test-Time Adaptation
Artificial Intelligence
Summary
Some AI models can adjust themselves when they face new or changing situations, but sometimes they get stuck by trusting their own mistakes. The authors introduce MASA, which helps models adapt more reliably by using a separate, fixed language-vision AI to provide clear descriptions of key examples. MASA uses these descriptions to guide the model’s adaptation without relying only on its own uncertain judgments. This way, models can better handle shifts in data like changes in style, viewpoint, or class imbalance during online use.
What this means in practice
- •For machine learning engineers: Enhance online model adaptation in changing environments by integrating semantic anchors from fixed multimodal models for more stable updates.
- •For computer vision developers: Improve performance of vision models under real-world distribution shifts by using semantic cues to guide test-time adaptation with limited data.
Authors
Zhenbin Wang, Lei Zhang, Lituan Wang, Yan Wang, Zhao Zhang, Wei Huang
Abstract
Wild test-time adaptation (WTTA) updates a source model online under small test batches, concurrent distribution shifts, and time-varying class imbalance. Most WTTA methods derive their adaptation signals, including predictive uncertainty, sample reliability, and local feature geometry, from the model being adapted. When the source model is unreliable under shift, these signals can reinforce its own errors, forming a self-referential loop. We introduce MASA (Multimodal-LLM-Anchored Semantic Adaptation), which complements model-internal evidence with structured semantic descriptions from a frozen multimodal large language model (MLLM). To limit inference cost, MASA queries the MLLM only for a small set of diverse, reliability-ranked anchors. The resulting descriptions capture the object family and nuisance factors such as style, viewpoint, and occlusion. MASA encodes these descriptions, propagates them to neighboring test samples, and stores the resulting visual-semantic information in an online prototype memory. Descriptor-aware retrieval from this memory provides an auxiliary target for lightweight adaptation of normalization-affine parameters. We evaluate MASA on the WTTA ImageNet-C benchmark under limited-batch, mixed-domain, and imbalanced-label-shift settings with ResNet and ViT backbones.