DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultimedia
AI summaryⓘ
The authors studied models that try to do both understanding and creating images or text using one shared system. They found that these two tasks want different kinds of internal information, which causes problems when combined in one model. To fix this, they developed a method called DIVA that splits the internal information into parts shared by both tasks and parts unique to each. This helps the model get better at both understanding and generation without the tasks interfering with each other. Their experiments showed clear improvements on both tasks.
Unified Multimodal ModelsInductive BiasRepresentation LearningVisual UnderstandingImage GenerationMulti-task LearningMutual InformationSelf-improved Post-trainingSemantic Embeddings
Authors
Renjie Lu, Xulong Zhang, Xiaoyang Qu, Shangfei Wang, Jianzong Wang
Abstract
Unified Multimodal models (UMMs) built on a single architecture have shown impressive performance in both understanding and generation. We identify a fundamental challenge that lies in inductive biases induced by distinct supervision signals: generation branch prefers high-fidelity, fine-grained representations capable of reconstruction, while the understanding favours semantically discriminative embeddings that remain invariant to task-irrelevant factors. Consequently, optimizing these complementary but non-equivalent objectives within a monolithic backbone leads to mutual impairment instead of enhancement. In this paper, we first analyze the root cause of this interference in unified backbones and reveal a complementary structure in their internal representations. Motivated by the observation, we propose DIVA, a self-improved post-training framework that transforms the representation divergence into interior synergy. By explicitly factorizing the visual representation into shared and unique components based on two complementary information flow, DIVA enables both the understanding and generation branches to achieve beneficial transferring while preserving the integrity of unique information from cross-flow interference via mutual information estimation. Despite its generality, our method consistently achieves improvements across visual understanding (+7.82%) and generation (+8.46%). The official code is available at: https://github.com/Jayyy-H/DIVA.