Vision language models recall rare classes better with cross-prompt inference
Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning
Computer Vision and Pattern Recognition
Summary
When vision-language models are trained on unbalanced data, they often forget how to recognize rare classes, doing worse than before training. The authors found that using a different descriptive prompt at inference time than the one used in training helps these models remember rare classes better. This effect happens because the model relies less on the parts changed during training and more on its original knowledge. Using this technique can improve detection of rare classes in several datasets without fully retraining the model.
What this means in practice
- •For machine learning engineers: Improve recognition of rare classes in vision-language models without extra training by using cross-prompt inference at test time.
- •For computer vision product teams: Enhance diagnostic or identification systems to better detect rarely seen categories by switching inference prompts from the finetuning prompt.
Authors
Emanuel Ben Baruch, Sivan Biham, Alon Shoshan, Yochai Zvik, Avi Ben-Cohen
Abstract
When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model's own zero-shot performance. In this regime, we uncover a surprising matched-prompt penalty whereby the standard choice of reusing the descriptive finetuning prompt at inference can yield lower rare-class recall than using a different descriptive prompt. We call this inference-time strategy cross-prompt inference. We evaluate diverse VLMs across six datasets and find that cross-prompt inference consistently recovers rare-class recall, with varying trade-offs in frequent-class recall. The effect reflects a train-eval prompt interaction rather than prompt quality alone, and does not arise with non-descriptive prompt changes. It is strongest when imbalance and visual confusability produce substantial rare-class suppression, and is weak or absent when such suppression does not occur. Using Tuning Contribution (TuCo), which decomposes a model's computation into pretrained and finetuning parts, we find that cross-prompt inference reduces reliance on finetuning-induced computation and shifts the model toward its pretrained component. Reducing that finetuning contribution also substantially reduces the cross-prompt advantage. These results show that rare-class knowledge suppressed by imbalanced finetuning can remain accessible under a different inference prompt.