Probabilistic prompt learning improves vision language model accuracy
ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Vision-language AI models like CLIP can identify new things using special prompts, but they struggle when given very few examples or when the images change a lot. The authors propose ProCAP, a new method that helps the model learn better by having the image and text parts talk more closely and update together in a safe way. ProCAP uses probabilities to avoid overfitting and adds an extra step to align images and words better. This approach works well on many different tests without changing the original model.
What this means in practice
- •For computer vision engineers: Improve few-shot image classification accuracy under changing data distributions using probabilistic cross-modal prompt learning without retraining the model.
- •For ai product developers: Enhance the robustness of vision-language features in products that must recognize new object categories with limited training data.
Authors
Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
Abstract
Pre-trained vision-language models such as CLIP can recognize new categories via prompting, but they often struggle when labeled data are scarce or the test distribution shifts. Prompt learning adapts only a small set of parameters while keeping the backbone frozen, yet many existing multimodal prompt learners couple the visual and textual branches weakly and can be brittle in low-shot regimes. We propose ProCAP, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine each other across prompt depth. To reduce overfitting under limited supervision, we parameterize prompt tokens with Gaussian means and variances and regularize them with lightweight KL and L2 penalties, and we further add a compact symmetric InfoNCE head that aligns cross-attended image features with class-level text representations in a shared low-dimensional space. Across few-shot base-to-novel generalization on 11 datasets, cross-dataset transfer, and domain generalization on ImageNet shift benchmarks, ProCAP achieves strong aggregate base-to-novel performance and competitive transfer performance while keeping the CLIP backbone unchanged.