Unified multimodal models improve by learning from their own generated images
Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models
Computer Vision and Pattern Recognition
Summary
Multimodal models combine understanding and creating images and text in one system, but usually treat these tasks separately. The authors developed a method called Generative Grounding Feedback that helps these models learn better by using their own generated images as feedback, without needing extra labeled examples. Their approach uses the model's own 'dreams'—images created from text prompts—and checks if the understanding and generation parts agree. This process helps the model improve making images from text and also slightly boosts how well it understands visuals.
multimodal modelstext-to-image generationvisual understandinggenerative feedbacklatent statepost-trainingself-evolvingcaptioningimage groundingpaired supervision
Authors
Ke Hao, Yuanzhi Liang, Tingxi Chen, Rui Li, Haibin Huang, Chi Zhang, Yun Gu, Xuelong Li
Abstract
Unified multimodal models integrate visual understanding and generation within a single network, yet the two capabilities are commonly optimized as separate tasks. We introduce Generative Grounding Feedback(GGF), a self-evolving post-training framework that uses only text prompts and the model's own visual experience. Given a prompt, the model first generates a visual ``dream.'' Flow-level feedback compares text-, image-, and repair-conditioned predictions at the same noisy latent state, transferring image-grounded generation directions to the prompt condition. Dream replay grounding replays this dream through captioning and re-imagination, training claim-level evidence to remain consistent across the replay while separating unrelated visual experiences. Jointly optimized, these two directions let generation provide visual grounding for understanding and understanding refine subsequent generation without paired image--text supervision. Experiments across unified models with different understanding--generation integration designs show consistent improvements in text-to-image generation together with modest gains in visual understanding.