Feature-space alignment improves deep vision model robustness without training
Technical note on: Zero-Training Feature-Space Alignment via Information Geometry
Computer Vision and Pattern Recognition
Summary
Deep vision models can struggle when the environment changes in ways they weren’t trained for, causing their accuracy to drop. The authors propose a new method called ZFGA that adjusts the model's internal feature representations at test time without changing the model itself or doing any training. This method works by measuring how the geometry of features changes between clean and shifted data and then aligning them to reduce errors. Their approach improves performance across several models and datasets while requiring only simple computations.
What this means in practice
- •For machine learning engineers: Improve image classification robustness in production systems facing real-world distribution shifts without extra model training.
- •For autonomous vehicle teams: Enhance perception model reliability under varying environmental conditions by aligning feature geometry at inference time.
Authors
Behraj Khan, Tahir Qasim Syed, Syed Ahmad Chan Bukhari
Abstract
Deep vision models often degrade under distribution shift. Test-time adaptation can improve robustness but typically requires iterative optimization, hyperparameter tuning, and multiple forward-backward passes. We propose Zero-Training Fisher Geometry Alignment (ZFGA), a closed-form method that improves robustness under covariate shift without modifying model parameters. ZFGA is based on the observation that distribution shifts distort feature-space geometry. It estimates the Fisher information matrix of the predictive distribution with respect to feature embeddings and applies a linear transformation that aligns test-feature Fisher geometry with a reference geometry computed from clean data. This provides a natural-gradient-inspired preconditioning step in feature space. We evaluate ZFGA on CIFAR-10-C and ImageNet-C using ResNet-50, DINO ViT-S/16, and CLIP ViT-B/32. ZFGA consistently improves over zero-shot inference across all three models, although it is not the strongest method for every model. Covariance whitening performs better on ResNet-50, while Fisher whitening is statistically indistinguishable from ZFGA on CLIP. Across six training-free and gradient-based alternatives (covariance whitening, Fisher whitening, TENT, T3A, LAME, and AdaNPC), ZFGA is the only method that does not substantially harm any of the three model families. The Fisher geometry distortion is also positively correlated with ZFGA gain (Pearson r = 0.366, p = 0.017), providing preliminary evidence that geometric misalignment contributes to robustness degradation. ZFGA requires only forward passes and matrix operations at inference time, offering a lightweight and deterministic alternative to optimization-based test-time adaptation.