Unified multimodal models improve image generation and understanding together
Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Models
Computer Vision and Pattern Recognition
Summary
Unified multimodal models often handle image understanding and image creation as separate tasks. The authors introduce a way for these models to use their own generated images to get better at both tasks at the same time without needing special paired training data. Their approach involves the model 'dreaming' a picture from text, checking that picture’s fit with the text, and then replaying this process to improve consistency. This self-feedback method helps the model get better at creating images from text and also modestly improves its ability to understand images.
What this means in practice
- •For ai developers: Improve text-to-image generation systems by integrating self-feedback mechanisms that boost image quality and consistency using only text prompts and model-generated visuals.$Commercial implications: Enables development of advanced image generation services with higher visual realism and alignment, appealing to industries like digital content creation.
- •For multimodal system engineers: Enhance unified models to balance visual understanding and generation more effectively without additional paired data, improving overall multimodal AI system robustness.
Authors
Ke Hao, Yuanzhi Liang, Tingxi Chen, Rui Li, Haibin Huang, Chi Zhang, Yun Gu, Xuelong Li
Abstract
Unified multimodal models integrate visual understanding and generation within a single network, yet the two capabilities are commonly optimized as separate tasks. We introduce Generative Grounding Feedback(GGF), a self-evolving post-training framework that uses only text prompts and the model's own visual experience. Given a prompt, the model first generates a visual ``dream.'' Flow-level feedback compares text-, image-, and repair-conditioned predictions at the same noisy latent state, transferring image-grounded generation directions to the prompt condition. Dream replay grounding replays this dream through captioning and re-imagination, training claim-level evidence to remain consistent across the replay while separating unrelated visual experiences. Jointly optimized, these two directions let generation provide visual grounding for understanding and understanding refine subsequent generation without paired image--text supervision. Experiments across unified models with different understanding--generation integration designs show consistent improvements in text-to-image generation together with modest gains in visual understanding.