Vision language models adapt faster using position aware layer queries
Position Aware Layer Queries for Test Time Training in Vision Language Models
Computer Vision and Pattern Recognition
Summary
Models that understand images and language sometimes struggle with unexpected or new types of images. The authors found a way to help these models adjust quickly at test time, using fewer computer resources. They do this by teaching a small network to learn important details inside the bigger model in just one pass, instead of many. This method improves accuracy on harder-to-classify images and works well on several types of vision and language models.
What this means in practice
- •For computer vision engineers: Speed up adaptation of image-language models to unusual images without retraining the entire system.
- •For machine learning system integrators: Improve accuracy and efficiency of language-vision models used for complex image understanding tasks like segmentation.
- •For software developers in augmented reality: Enable AR applications to quickly adjust vision-language models for new environments improving robustness in real-time scenarios.$Commercial implications: This method allows creation of AR products that adapt visual understanding on the fly, enhancing user experience and reliability.
Authors
Rajat Modi, Priyank Pathak, Xin Liang, Yogesh Singh Rawat
Abstract
Test-Time Training (TTT) adapts models to incoming test samples (e.g. out-of-distribution, (OOD)) when conventional fine-tuning is infeasible. Existing TTT methods for Vision-Language Models (VLMs) create supervision from several augmented views, each requiring forward (and often backward) passes through the entire VLM, incurring substantial computational cost. We observe that one forward pass with all the intermediate layer outputs already yields far more signal than the final embedding from all augmentations. We introduce Layer Query Network (LQN), a lightweight approach that can adapt a frozen VLM (teacher) in a single forward pass of the VLM via a small model (student). LQN uses Position-Aware Distillation (PAD) to mimic the teacher VLM's intermediate-layer spatial tokens by querying spatial coordinates of intermediate tokens. LQN additionally relies on Location Consistency Regularization (LCR), a self-supervision technique, replacing expensive O(H x W) image augmentation with O(1) coordinate sampling. Integrating these, LQN i) adapts and improves zero-shot CLIP ViT-B/16 by 9.8% Top-1 on OOD ImageNet, ii) outperforms the previous best GS-Bias on fine-grained classification by 3.9% Top-1, iii) achieves faster convergence than TPS for CLIP ResNet-50 (47 mins vs 55 mins), iv) generalizes adaptation to VLMs like SigLIP, EVA-CLIP, and CoCa, and lightweight students like MLP, ResNet, VGG, and v) extends to panoptic, instance, and semantic segmentation.