Vision-of-thought adds visual planning layer for clear image generation
VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Generating images from text often involves a model that tries to turn words into pixels directly, which can be hard to understand or control. The authors introduce a new method that adds a middle step: a kind of visual thinking where the model creates tokens that represent objects and layouts before making the final picture. They train this step so these tokens make sense to both language and image parts of the system. This helps the model produce images that better match the meaning of the text and makes the image creation more interpretable and easier to manage.
What this means in practice
- •For ai developers: Create image generation tools that give more control and understandable intermediate results linking language and vision.
- •For computer graphics teams: Develop interpretable image synthesis workflows by inserting discrete visual planning before rendering pixels.
Authors
Jingxiang Sun, Chao Liao, Zhengxiong Luo, Chaorui Deng, Chen-lin Zhang, Junke Wang, Ceyuan Yang, Haoqi Fan, Weilin Huang
Abstract
Current text-to-image systems typically employ a "text encoder plus diffusion decoder" paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals. In this paper, we propose Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs). Instead of treating VLMs merely as text encoders, we use them as multimodal planners that generate discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels. We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses. These objectives make the tokens semantically readable by the VLM while preserving the visual information needed for generation. Experimental results demonstrate that VoT improves semantic alignment and provides a structured interface for interpretable and controllable generation.