Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models
Computer Vision and Pattern Recognition
Summary
The authors study a way to adapt Vision-Language Models (VLMs) to new domains without needing original source data or models, which is called Source-Fully-Free Domain Adaptation (SFF-DA). They find that two types of semantic misalignment, called Dual Semantic Drift, cause problems during adaptation. To fix this, the authors propose DSSG, a method that balances maintaining overall category stability with learning new domain details using two streams: one focusing on captions and one on class anchors. They improve this with DSSG-PAC, which speeds up adaptation by reusing calibrated class anchors. Their experiments show these methods work better than previous approaches while saving time.
Authors
Weiwei Xiang, Shun Peng, Guangyi Xiao, Hao Chen, Lei Yang
Abstract
Source-Fully-Free Domain Adaptation (SFF-DA) has emerged as a strategic paradigm to adapt Vision-Language Models (VLMs) without any access to source data or task-specific source models. However, we identify a critical Dual Semantic Drift that hinders this process: static drift arising from the rigidity of fixed class embeddings, and dynamic drift stemming from the divergence of generated captions, causing severe semantic misalignment that intensifies the stability-plasticity dilemma. To address this, we propose DSSG (Dual-Stream Semantic Guidance), an end-to-end framework that reconciles fine-grained plasticity with global stability. Our core contribution is the Dual Semantic Guidance (DSG) module, which integrates a caption stream for domain-specific knowledge with a class-anchor stream to anchor global categorical consistency. Furthermore, a Dynamic Cross-Modal Knowledge Distillation (CMKD) module is introduced to leverage the evolving teacher distribution for calibrating teacher-student consistency. Building upon DSSG, we further introduce Prototype Anchor Calibration (PAC), yielding DSSG-PAC, which periodically calibrates prototype anchors and caches them until the next calibration. This design reduces redundant text-side computation while preserving the adaptability of class guidance to the evolving text space. We further establish SFF-DA risk bounds that relate student risk to semantic-teacher quality and teacher--student discrepancy. Extensive experiments demonstrate that DSSG consistently outperforms current state-of-the-art methods across multiple benchmarks, while DSSG-PAC largely preserves its adaptation performance with 18.9% lower total adaptation time. The code is available at https://github.com/mrmenand/DSSG.