A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
2026-08-26 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how large multimodal language models (MLLMs) can keep learning from continuous streams of unlabeled data without forgetting previous knowledge. They found that paying attention to how much each token depends on visual information (visual dependence) is important to prevent losing old skills and to learn new tasks better. They propose a method called Visual Dependence-Aware (VDA) with two parts: one that carefully maintains old visual knowledge during updates, and another that helps focus learning on new visual information. Their experiments show this approach balances remembering old tasks while adapting to new ones.
Multimodal Large Language ModelsUnsupervised Post-TrainingVisual DependenceCatastrophic ForgettingOptimal TransportCross-Modal LearningPlasticity vs StabilityVisual GroundingContinual LearningStreaming Data
Authors
Kaichen Li, Zhilin Zhu, Jianhao Huang, Zhengqin Lai, Baochen Xiong, Zibo Shao, Yaguang Song, Linhui Xiao, Xiaoshan Yang, Changsheng Xu
Abstract
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.