Self correction improves consistent image and text generation together
Self-correction Optimization for Interleaved Multimodal Generation
Computer Vision and Pattern Recognition
Summary
Creating a story that mixes pictures and words in the right order is hard because the computer needs to understand and generate both well at the same time. The authors propose a new method called self-correction optimization (SCO) that helps the computer fix itself while generating these mixed stories without needing extra training. SCO encourages the story to stay consistent over time and make sure characters or objects look the same throughout the story. Their tests show that SCO helps make these picture-and-text stories more coherent and realistic, and it also works to improve videos and robot actions.
What this means in practice
- •For multimedia developers: Create consistent stories or conversations mixing images and texts that maintain character and scene coherence without costly retraining.
- •For robotics teams: Improve simulations or planning for robot tasks by better modeling sequences of visual and physical changes over time.
Authors
Xin You, Zhiwei Ning, Zukai Chen, Minghui Zhang, Xuanke Shi, Hanxiao Zhang, Jingsong Liu, Jie Yang, Quan Wang, Yun Gu
Abstract
Multimodal large language models (MLLMs) have made significant progress in visual understanding and generation. However, generating interleaved image--text content remains challenging, as it requires tightly integrated multimodal understanding and generation capabilities. Although existing MLLMs provide promising solutions, most rely on additional training with augmented data, which is computationally expensive and remains limited in preserving visual subjects, temporal consistency, and physical plausibility. In this work, we propose self-correction optimization (SCO), an effective training-free method for consistent interleaved generation. SCO treats the classifier-free guidance update as a reference and performs minimal self-correction under two complementary constraints, including new-event and state-preserving constraints. Specifically, the new-event constraint promotes temporal consistency across image--text sequences, while the state-preserving constraint maintains the coherence of visual subjects throughout subsequent generation steps. Experiments on challenging interleaved multimodal generation benchmarks demonstrate significant improvements in temporal coherence and visual-subject preservation. Furthermore, SCO can be extended to video generation and improves the modeling of physically grounded processes, including robot manipulation and long-horizon handcrafting.