Vision language models improve generalization with dual prompt routing
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models can learn to recognize objects they have seen before but often struggle with new, unseen objects. The authors propose a new method called PETR that uses two types of prompts: one focused on classes seen during training and another that helps generalize to unseen classes. Instead of training a complex router, PETR uses statistics from the training data to decide which prompt to use for each new image, making the decision transparent and stable. Tests on many datasets show PETR improves performance on both known and unknown objects compared to earlier methods.
What this means in practice
- •For machine learning engineers: Improve accuracy of vision-language systems on both classes seen in training and new, unseen classes without retraining large models.
- •For computer vision developers: Build adaptable recognition systems that automatically select best prompt for each image to handle diverse visual tasks.
Authors
Weihan Cai, Hao Tan, Xinping Gao, Shibiao Xu, Jun Wan
Abstract
Prompt learning efficiently adapts vision-language models (VLMs) to downstream tasks, but gains on seen classes often come at the expense of generalization to unseen classes. To address this limitation, we propose prompt ensembling with training-free routing (PETR), whose key innovation is a carefully designed dual-prompt architecture: two complementary prompts are learned from different data and objectives to emphasize seen class discrimination and unseen-class generalization, respectively. During training, both prompts are fine-tuned using a shared frozen CLIP backbone, and statistical information is collected from the training set logits. At inference time, we determine the similarity of each test sample to seen data, and route the sample to the most appropriate prompt branch. To the best of our knowledge, this is the first prompt tuning framework that performs training-free adaptive routing based on statistical similarity. This design provides an interpretable routing signal and avoids common MoE-style routing pathologies, such as router training instability and load imbalance. Extensive experiments on 11 benchmark datasets demonstrate that our framework consistently outperforms previous methods on both seen and unseen classes, achieving new state-of-the-art results.