Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification
2026-06-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how to improve machine learning models when they face new, different data without labels, a situation called Test-Time Adaptation (TTA). They find that just minimizing uncertainty (entropy) can lead to many equally good but very different solutions, causing instability. To fix this, they propose exploring multiple possible solutions at once, like testing several paths simultaneously, which helps the model stay reliable. Their method works with existing TTA approaches and shows better and more stable performance on hard tests.
Test-Time AdaptationEntropy MinimizationDistribution ShiftUnderspecificationDecision BoundaryPseudo-likelihoodParticle-based MethodsMulti-hypothesis InferenceModel RobustnessUnlabeled Data
Authors
Afshar Shamsi, Xiao-Yu Guo, Hamid Alinejad-Rokny, Arash Mohammadi, Damien Teney, Ehsan Abbasnejad
Abstract
Test-Time Adaptation (TTA) seeks to improve model robustness under distribution shifts by adapting parameters using unlabeled target data. However, in the absence of supervision, entropy-based adaptation is fundamentally underconstrained: multiple distinct parameter updates can achieve similarly low entropy while inducing drastically different decision boundaries. This phenomenon, known as underspecification, renders standard TTA brittle and prone to collapse into spurious modes. In this work, we reinterpret TTA through a posterior-inspired lens induced by entropy minimization, where low-entropy solutions define a pseudo-likelihood over parameters. Instead of committing to a single point estimate, we introduce a particle-based diversification framework that explores multiple plausible adaptation trajectories simultaneously. Our method can be viewed as a structured exploration of multiple plausible adaptation solutions, implemented through multi-level diversification at the output, parameter, optimizer, and input levels. Crucially, the framework acts as a plug-and-play wrapper compatible with existing TTA methods. Extensive experiments on challenging benchmarks demonstrate consistent gains in stability and robustness, achieving improvements of 3-4% under mixed shifts, 2-3% with batch size one, and 1-2.5% under label shifts, outperforming state-of-the-art baselines. Our results suggest that treating TTA as a multi-hypothesis inference problem, rather than a single-point optimization task, is key to mitigating underspecification and enabling reliable real-world deployment.