Compatible routing improves continual learning in multimodal models
CART: Compatible Routing for Multimodal Continual Instruction Tuning
Artificial Intelligence
Summary
Multimodal AI models that understand images and text need to keep learning new skills without forgetting old ones, which is tricky. The authors found that updating just one part of the model while freezing another causes problems, making the model forget. They designed a method called Compatible Routing (CART) that updates both parts of the model in a coordinated way, using multiple mini-experts working together. This helps the model learn new tasks better without losing what it already knows.
What this means in practice
- •For ai system builders: Develop improved multimodal AI systems that can update with new visual and language tasks sequentially without losing prior knowledge.
- •For software engineers in autonomous vehicles: Enhance vehicle perception modules by continually integrating new multimodal instruction data while maintaining stable model performance.
Authors
Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou
Abstract
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert consolidation, which merges experts exhibiting similar relevance profiles across samples within each module. Extensive experiments demonstrate that CART achieves state-of-the-art performance.