Vision of thought adds clear visual planning to image generation systems
VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Text-to-image systems usually transform words directly into pictures using complex math, but this process is hard to understand or control. The researchers created a new step called Vision-of-Thought (VoT) that sits between language understanding and image creation. VoT breaks down the process into clear visual ideas like objects and layouts before making the final image. This helps make image generation more understandable and allows better alignment between what the text means and what the picture shows. In tests, VoT improved how well the generated images matched the original text and made the system easier to guide.
text-to-image generationvision-language modelsdiffusion modelsdiscrete tokensvector quantizationsemantic alignmentmultimodal representationimage synthesislatent spaceinterpretability
Authors
Jingxiang Sun, Chao Liao, Zhengxiong Luo, Chaorui Deng, Chen-lin Zhang, Junke Wang, Ceyuan Yang, Haoqi Fan, Weilin Huang
Abstract
Current text-to-image systems typically employ a "text encoder plus diffusion decoder" paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals. In this paper, we propose Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs). Instead of treating VLMs merely as text encoders, we use them as multimodal planners that generate discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels. We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses. These objectives make the tokens semantically readable by the VLM while preserving the visual information needed for generation. Experimental results demonstrate that VoT improves semantic alignment and provides a structured interface for interpretable and controllable generation.