Dual-mode learning improves visual language model class updates
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Adding new visual classes to AI models can overwrite old knowledge or miss chances to share what the model already knows. The authors developed a method called DuLBE that smartly balances reusing existing visual information while allowing learning new tasks without interference. It connects visual patterns and text descriptions in a special way to better combine what the model sees and reads. Their tests show DuLBE keeps the model efficient and improves learning new classes without forgetting old ones.
What this means in practice
- •For computer vision engineers: Enhance continual learning systems to add new image classes without sacrificing past knowledge or requiring stored examples.
- •For natural language processing developers: Build vision-language models that better combine text and visual features to improve classification during incremental learning.
Authors
Chiyuan He, Zihuan Qiu, Fanman Meng, Chao Wang, Liangjiang Chen, Linfeng Xu, Qingbo Wu, Hongliang Li
Abstract
Benefiting from transferable visual-textual alignment, CLIP has been widely adopted for class-incremental learning (CIL). However, existing learners either repeatedly update components shared across tasks, leading to knowledge overwriting, or overly isolate new-task updates, hindering the reuse of CLIP's transferable knowledge and limiting plasticity. Moreover, the text-based or bimodal classifier designs still fail to effectively integrate complementary information from the visual and textual modalities. To address these challenges, we introduce DuLBE, which couples dual-mode low-rank learning with a bridge-prototype ensemble classifier for exemplar-free CIL. DuLBE allocates two visual low-rank update modes according to the gradient demand and uses gradient routing to coordinate them: a compact and rewritable shared mode is selected from historically occupied visual directions to reuse transferable knowledge, while residual modes provide low-interference channels for task-specific variations. Building on the resulting stable inter-modal structure, we further construct geodesic bridges between visual prototypes and text embeddings on the unit hypersphere, and ensemble reliable bridge prototypes to compensate for the modality-gap limitations of textual decision boundaries. Extensive experiments under multiple settings show that DuLBE achieves state-of-the-art CIL performance while retaining the high parameter efficiency of low-rank tuning.