Test-time adaptation improves AI accuracy by steering frozen network parts
SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representations
Machine LearningComputer Vision and Pattern Recognition
Summary
AI models often struggle when they see data that’s a bit different from what they were trained on. This paper shows a way to help AI adjust on the fly by changing small parts inside the model without retraining the whole thing. Instead of changing the main model, the method nudges internal representations to better fit new data using small, learnable modules. This approach works well across different types of AI models and keeps the original skills intact while improving performance on new, shifted data.
What this means in practice
- •For computer vision engineers: Improve image classification models to adapt to changing real-world conditions without retraining or access to original training data.
- •For autonomous vehicle developers: Enhance perception systems to adjust to unexpected visual conditions on the road through lightweight real-time model adaptations.
Authors
Muhammad Sudipto Siam Dip, Ali Etemad
Abstract
Test-time adaptation (TTA) addresses distribution shift using only unlabeled test data. Existing methods typically adapt pretrained models by updating their parameters, limiting both what is adapted and where adaptation can occur within the network. We instead keep the pretrained network frozen and steer its intermediate representations. We introduce SPeaR (Steering Primitive for Realigning Representations), which inserts lightweight learnable modules at stage boundaries and optimizes them directly from the test stream, requiring neither source data nor supervised warm-up. Each primitive is optimized using a gated objective that reduces uncertainty only when adaptation is beneficial, along with a diversity regularizer to prevent collapse, and a multi-depth anchor to stabilize adaptation. We show that steering early representations is the most effective strategy, and that the same primitive transfers across convolutional and Transformer architectures. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-C, SPeaR consistently matches or outperforms methods that adapt orders of magnitude more parameters, remains robust across a wide range of batch sizes, and preserves source-domain performance during continual adaptation.