Vision language models plan image descriptions step by step
How VLMs Plan What's Next
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
When describing images, vision-language models choose what to mention one object at a time, but it wasn’t clear how they decide that order. The authors explain that the models use special parts called binding heads and gaze heads to point to image regions, keep track of what’s been described, and decide what to name next. They tested these parts by changing the model and seeing how it affected performance, confirming the process works consistently across different images, languages, and model sizes. This helps us understand how these models plan multi-step answers from images.
What this means in practice
- •For image captioning developers: Improve image captioning systems by incorporating explicit planning steps that choose objects sequentially to enhance descriptive accuracy and coherence.
- •For multilingual ai teams: Build vision-language models that plan token outputs consistently across languages without redesigning core components.
Authors
Rohit Gandikota, Arnab Sen Sharma, Adam Belfki, David Bau
Abstract
When a vision-language model describes an image, it names one object at a time and it rarely repeats itself, yet how the model plans this order is far from obvious. We characterize the mechanism a VLM uses to decide where to look before it looks. The mechanism has four main parts. First, binding heads match the relevant text tokens against the image and resolve them into pointers to the matched image region. Second, the same binding heads bind each written answer to the region it came from, and the model turns it into a progress signal: a cursor, the position in the image up to which the list has been written. Third, a select step chooses the nearest unwritten region after the cursor, or stops. Finally, gaze heads execute the chosen pointer and return a concept of what is drawn there, which language heads and the late MLPs turn into a word in the language of the question. We validate each part of this mechanism through causal patching in Qwen3-VL models on a synthetic visual retrieval task, and test our mechanisms against alternative hypotheses. We further show that the mechanism holds across input formats, natural photographs, and languages, and that its binding, relay, and gaze heads appear at every model size from 2B to 32B. This work allows us to investigate how a VLM turns a question and an image into a sequence of answers, layer by layer and token by token. More broadly, it opens a way to study how models plan over complex multi-step tasks with long rollouts. Our project is available at https://plan.baulab.info/