AI summaryⓘ
The authors study how well Vision-Language Models adapt to new tasks when given only a few labeled examples, without extra training. They point out that current tests often assume data has balanced classes, which isn't always true in real life. To tackle this, they create a new benchmark that includes imbalanced class distributions and varying numbers of classes, finding that existing methods perform worse under these conditions. To improve this, they propose PRiSM, a method that regularizes class prototypes to keep classes distinct and aligned with initial results, boosting performance especially when classes are unbalanced or numerous. They also design an efficient optimizer to solve the PRiSM method effectively.
Vision-Language Modelsfew-shot learningclass imbalanceDirichlet samplingclass prototypesregularizationblock Majorize-Minimize optimizerGershgorin circle theorempairwise inter-class distancessupport-feature alignment
Authors
Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz, Ismail Ben Ayed
Abstract
Training-free few-shot adaptation methods have gained significant attention recently in the context of Vision-language Models (VLMs). Yet, current benchmarks rely on strong assumptions about the statistics of the adaptation data, e.g., class balance. We question these simplifying assumptions and introduce a more realistic benchmark that varies both the levels of class balance and the effective number of classes in few-shot tasks via Dirichlet sampling. Surprisingly, under our setting, we observe substantial drops in the performances of state-of-the-art methods, more so when the number of labeled samples increases. To mitigate this, we introduce PRiSM, a class-prototype regularization that can be deployed as a plug and play module on top of any existing baseline method, significantly improving performances. Our method optimizes a novel multi-term loss, which includes a regularizer maximizing inter-class pairwise distances, along with additional terms promoting support-feature alignment and fidelity to the baseline prototypes. Furthermore, we introduce an effective and computationally efficient block Majorize-Minimize optimizer for our objective. More specifically, we derive a valid blockwise Lipschitz constant (i.e., a bound on the Hessian's spectral norm), which can be computed efficiently via the Gershgorin circle theorem. Extensive experiments show that PRiSM improves several training-free baselines, with large gains when dealing with severe class imbalance and high numbers of classes.