Visual generation training improves unified multimodal models knowledge
Can Visual Generation Expand What Unified Multimodal Models Know?
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Some AI models learn from both images and text together. The paper looks at whether training these models to create images helps them understand things better than just reading text. The authors find that learning to generate images does add new knowledge that can sometimes be accessed through text answers. They also suggest ways to improve how this image-learned knowledge can be used in text questions. This shows that teaching AI with pictures as well as words can expand what the AI knows.
What this means in practice
- •For ai model developers: Improve multimodal AI training by adding image generation tasks to expand the models’ knowledge beyond text-only data.
- •For natural language processing teams: Enhance question-answering systems’ performance by incorporating training methods that connect image generation with textual knowledge access.
Authors
Wenkai Li, Wenyi Hong, Zhuoyi Yang, Jie Tang
Abstract
Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.