Vision language models improve learning tasks without retraining
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models that understand images and text can lose accuracy when learning new tasks over time, especially without retraining on old data or knowing task details. The authors identify three main problems these models face and introduce a new method called COMET to address them. COMET uses a mix of expert components and special distillation techniques to preserve knowledge, coordinate experts, and manage model capacity effectively. Their approach keeps tasks separated during training but merges experts when helpful, improving continual learning performance.
What this means in practice
- •For machine learning engineers: Improve deployment of vision-language models in environments where new tasks arrive sequentially without retraining or access to old task data.
- •For robotics developers: Build robots that recognize objects and scenes continually without needing to retrain or know the context of tasks in advance.
Authors
Seungyoon Woo, Keonhee Park, Gunhee Kim
Abstract
Vision-language models (VLMs) such as CLIP retain zero-shot recognition ability, but their open-world use often requires task-agnostic continual learning: tasks arrive sequentially, past data are unavailable, task identity is unknown, and the backbone cannot be retrained. We argue that this setting fails VLMs through three coupled pressures beyond the stability-plasticity tradeoff: (1) modal-consistency drift, (2) representation-space interference, and (3) per-task capacity shortfall. We propose COMET (Collaborative Mixture-of-Expert Transfer) to resolve these three issues respectively as follows. A frozen teacher provides only a feature reference on a shared image-text pool; Tri-Affinity Distillation preserves image-image, text-text, and image-text geometry; and Student-Aware Cross-Expert Distillation aligns new experts with compatible prior experts so routing errors degrade gracefully. We design COMET to keep capacity and inference separate: a contextual bandit keeps or merges full per-task experts using post-training signals, while a reconstruction router selects sparse experts at test time without teacher semantics. On the X-TAIL benchmark, we show that each proposed component improves continual learning performance. COMET shows that continual vision-language adaptation can preserve multimodal geometry, coordinate accumulated experts, and allocate capacity jointly when feature reference, expert shaping, routing, and capacity control are separated by construction.