Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how vision-language models (VLMs) figure out spatial relationships between objects. They found that these models first roughly identify objects before reasoning about how those objects relate in space, but they don’t need exact object boundaries to do this. Different parts of the model handle object detection and spatial reasoning separately, though they share some early processing steps. This work helps explain how these models connect recognizing objects to understanding spatial relations.
Vision-language modelsSpatial reasoningObject groundingToken ablationAttention headsLayer-wise probingCausal mediation analysisBounding boxesSpatial relations
Authors
Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li, Zhixiang Lu, Haolin Yang, Imran Razzak, Yutong Xie
Abstract
Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.