Model improves image recognition on unseen domains without training data
Zero-Shot Domain Generalization via Domain Prompt Learning
Computer Vision and Pattern Recognition
Summary
Recognizing objects in new settings without retraining is hard because models usually need training examples from those environments. The authors present a method that uses a big vision-language model to understand new domains without needing any training images from those domains. Their approach learns style information via text prompts and then refines these styles by looking at unlabeled visual data during testing. This helps the system recognize new object types in new visual environments better than previous methods.
What this means in practice
- •For computer vision engineers: Improve image recognition models to work well on unseen environments without accessing labeled training data from those environments.
- •For machine learning platform developers: Integrate zero-shot domain adaptation methods to support deployment of vision models across diverse real-world scenarios without costly data collection.
Authors
Shiming Chen, Shuhuang Chen, Salman Khan, Fahad Shahbaz Khan
Abstract
Domain generalization (DG) aims to improve the generalization of the model to any unseen domains. Targeting this goal, existing DG methods mainly focus on learning domain-invariant features or prompts from the given source domains. However, these methods rely on source-domain data, and collecting and annotating large-scale single- or multi-source datasets is often infeasible, limiting the practical applicability of DG. To tackle this challenge, we propose {\underline{Do}}main \underline{P}rompt Learning (DoP), a method built on a large vision-language model (e.g., CLIP) for zero-shot domain generalization, which recognizes unseen classes in unseen target domains without access to source-domain data. DoP essentially learns the target domain knowledge via prompt both in text and visual modalities to overcome the domain shift in text-visual modalities under the test-time adaptation formulation. Specifically, DoP first takes text domain learning (TDL) to simulate various distribution shifts by learning diverse style tokens implicitly in the text space with inter-style diversity loss and intra-style consistency loss. Subsequently, DoP employs visual domain learning (VDL) to further refine the target-domain-relevant style token generated by TDL by making use of the test visual data with self-supervised learning. Extensive experiments on popular DG benchmark datasets (e.g., PACS, VLCS, OfficeHome, and DomainNet) and ImageNet show that our DoP consistently outperforms the state-of-the-art DG approaches. Codes are available at: https://github.com/shiming-chen/DoP.