Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
2026-06-02 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors describe a new method called Imaginative Perception Tokens (IPT) that helps vision-language models better understand spaces they can’t fully see by imagining what might be observed from different viewpoints. They created three tasks and datasets to test this ability, focusing on perspective taking, path tracing, and counting objects from multiple views. Their approach improves the models’ spatial reasoning more than other methods, like using language-based reasoning steps, and works well without needing to generate images during testing. IPT also produces understandable intermediate steps that show what the model is imagining, leading to better overall performance.
vision language modelsspatial reasoningimaginative perceptionperspective takingpath tracingmultiview countingintermediate representationssupervisionchain of thoughtgeneralization
Authors
Mahtab Bigverdi, Lindsey Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dangjoo Kim, Zelun Luo, Linda Shapiro, Ranjay Krishna
Abstract
Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation. We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input. To study this capability, we formulate three tasks, Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC), and construct datasets of approximately 20K examples with ground truth imaginations, answers, and evaluation benchmarks. Using the unified VLM BAGEL as the backbone, IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training, even without generating images at inference time. On MVC, IPT improves accuracy by 3.4% and achieves competitive performance with strong closed-source models on PT. We further find that combining IPT and label-only supervision yields additional gains, whereas textual chain of thought can substantially degrade performance, suggesting a modality mismatch when spatial computation is forced through language. Overall, IPT provides a principled supervision signal for reasoning about unobserved spatial structure, improving generalization while producing interpretable intermediate representations.