Layer ensemble improves few-shot image classifier without training
LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Vision-language models like CLIP are used to identify images, but usually only their final layer’s output is used to classify new examples. The authors found that intermediate layers also contain useful information, but their raw scores can’t be combined easily. They developed LE-GDA, a method that standardizes and combines information from multiple layers without extra training. This approach improves few-shot learning accuracy on many datasets and works well even with very limited examples. It shows that using more layers inside CLIP boosts performance when adapting the model to new tasks quickly.
What this means in practice
- •For computer vision engineers: Improve few-shot image classification accuracy by combining multiple model layer outputs without retraining.
- •For machine learning platform developers: Incorporate training-free multi-layer ensemble classifiers to quickly adapt pre-trained models to new image recognition tasks.
Authors
Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu, Jakin Mukendi Kabongo, David Mayoya Kutangila, Pescie Honore Kinkani Mampuya, L'Or Wolo Sieka
Abstract
Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.