Quantizer-aligned recalibration improves quantized vision transformer accuracy

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Vision transformers are models used for image tasks that can run on small devices by using quantization, which shrinks their size but can also make them less accurate when images change a bit from the training data. The authors found that existing methods to fix this problem either slow things down or don’t fully restore accuracy. They propose a new way to adjust the model’s internal settings without changing its parameters or needing extra processing. This adjustment, called Quantizer-Aligned Recalibration, improves accuracy and speed while using very little extra memory.

What this means in practice

  • For embedded device engineers: Improve image model robustness on edge devices by recalibrating quantized activations without extra computation or memory use.
  • For mobile app developers: Deliver more accurate real-time vision features on smartphones using single-pass, no-backprop quantizer recalibration techniques.

Authors

Hyeongheon Cha, Young D. Kwon, Sung-Ju Lee

Abstract

Post-training quantization is a standard route to fitting vision transformers (ViTs) into edge compute and memory budgets, yet quantized models become especially brittle under distribution shift. Test-time adaptation (TTA) addresses such shifts without labels, but most existing approaches are poorly aligned with the constraints of quantized inference. Prevailing TTA methods recover accuracy through backpropagation, while backprop-free methods often still incur overhead from extra forward passes or parameter updates, and lightweight feature- or logit-level methods recover only part of the loss. Across these approaches, a quantization-specific failure mode that amplifies the drop is not directly targeted: under shift, activations occupy frozen quantizers' calibrated ranges differently, distorting their code distribution. We propose Quantizer-Aligned Recalibration (QuAR), a single-pass TTA method tailored to quantized ViTs that neither backpropagates nor updates any model parameters. QuAR recalibrates activations at the input to a frozen quantizer, mapping the test stream's running per-channel statistics back toward the source calibration. On ImageNet-C with ViT-B, QuAR achieves the highest mean accuracy among state-of-the-art backprop-free TTA methods at 3-, 4-, 6- and 8-bit weight/activation precision, outperforming the strongest baseline by 2.28 points at 8 bits and 4.00 at 3 bits, with 46% lower latency and a memory overhead of only 0.17 MB (0.01% of peak inference memory). Analysis and diagnostics trace the gain to a reduced per-channel mismatch at these quantizers, which restores the code distribution the baselines leave unchanged or distort further. A single fixed configuration remains ahead across continual streams, non-i.i.d. label shift, seven out-of-distribution suites, and three other backbones.