Layerwise adaptive mixing boosts few-shot vision language accuracy
LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
Computer Vision and Pattern Recognition
Summary
Identifying things with only a few example pictures is hard because many models use only one part of their info. The authors show that by mixing info from several layers of a visual model, more helpful clues are found. This method, called LAMP, combines these clues without needing extra training and improves how accurately computers recognize images from few examples. It works with different kinds of models and on many datasets, making it easier to adapt vision and language systems quickly.
What this means in practice
- •For machine learning practitioners: Improve few-shot image classification accuracy without retraining models by combining multi-layer features from frozen vision-language encoders.
- •For computer vision engineers: Develop vision-language applications that adapt quickly to new tasks with minimal labeled data by leveraging layerwise prediction mixing methods.
Authors
Zhipeng Ye, Feng Jiang, Qiufeng Wang, Hao Li
Abstract
Few-shot adaptation often relies on an encoder's final global representation, leaving complementary information in intermediate layers underused. Combining these features is challenging because their dimensions can differ and their usefulness varies across tasks. We propose LAMP, Layerwise Adaptive Mixing of Predictors, a training-free framework that combines global and regional information from multiple encoder layers. A single forward pass per view supplies all descriptors. Global and support-calibrated regional kernel predictors independently fit the same label residuals. Labeled validation data select their mixture in class-score space, preserving native feature dimensions without learned projections. LAMP-F further adapts text-anchored class prototypes, using a cached response to update the fixed-kernel residual predictions exactly. Experiments cover eleven datasets and ViT and ResNet backbones. LAMP achieves state-of-the-art mean accuracy among the compared published training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots. Averaged across five support budgets, LAMP improves over a paired global-feature-only baseline by 1.14 and 1.59 percentage points on ViT-B/16 and ResNet-50; LAMP-F raises these gains to 1.96 and 2.54 points. LAMP-F reaches 16-shot accuracies of 83.57% and 79.10%, respectively. Controlled experiments identify multi-layer integration as the main source of improvement and examine pooling, calibration, validation labels, and computational cost. These findings show that combining layer-specific predictions can exploit complementary features from frozen encoders to improve few-shot classification.