LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine LearningMultimedia
AI summaryⓘ
The authors studied how to adapt a model trained on one dataset to work well on a new, unlabeled dataset without needing the original data. They introduced a method using big foundation models, like vision-language models, to help guess labels for new data, including classes the model hasn't seen before. Their approach checks for different kinds of label changes and refines guesses by combining knowledge from both the foundation models and original trained models. Tests showed their method works better than previous ones on various challenges. They also shared their code for others to use.
Source-free universal domain adaptationFoundation modelsVision-language modelsLabel shiftCovariate shiftPseudo-labelingGaussian mixture modelZero-shot learning
Authors
Jing Li, Pan Liu, Meng Zhao, Wanli Xue, Yanhong Yang, Xu Cheng, Fan Shi, Jianhua Zhang, Qinghua Hu, Shengyong Chen
Abstract
Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.