CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
2026-06-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors tackle the problem of teaching a multimodal language model to learn new tasks continuously without forgetting old ones. They propose a method called CRAM, which separates task-specific knowledge into independent modules to avoid interference between tasks. CRAM also smartly adds only the needed new parameters based on task difficulty and reuses existing knowledge when possible. Their experiments show that CRAM works better than previous approaches in balancing learning new tasks and remembering old ones.
Multimodal Large Language ModelsContinual LearningInstruction TuningCatastrophic ForgettingParameter EfficiencyModular NetworksAdaptive-rank InstantiationRouting MechanismOrthogonality Penalty
Authors
Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou
Abstract
Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.