Geometric encoding improves spatial understanding in vision language models
Geometric Encoding for Spatial Reasoning in Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Vision-language models can recognize objects in videos but struggle to understand where things are and how far apart they are. The authors created a way to extract detailed spatial information like object size, distance, and room layout from videos and turn it into a clear code. This code is given to the models as extra information, helping them make better sense of the video’s spatial details. This method improved performance on tasks involving measurements and counting without needing extra training.
What this means in practice
- •For video analytics developers: Integrate explicit spatial encoding to improve metric reasoning and object tracking in video analysis tools.
- •For robotics engineers: Use explicit geometric codes from monocular video to aid robots in spatial understanding and scene reasoning without extra training.
Authors
Antonio Jun, Haoshui Yu, Zhengyi Lu, Huirong Fu, Yao Qiang
Abstract
Vision-Language Models (VLMs) are far more reliable at recognizing what appears in a video than at reasoning about its spatial and temporal properties, such as metric distances, object dimensions, and consistent object identities across frames. We present Geometric Code, a perception-to-geometry pipeline that computes explicit spatial structure from video and supplies it to VLMs as context to augment reasoning. A perception layer segments and classifies objects and recovers depth, camera pose, and intrinsics from monocular RGB video. A deterministic geometric engine then back-projects, merges, and cleans these outputs into a spatial code, including per-object positions, dimensions, counts, inter-object distances, appearance order, and room geometry. The code is serialized into VLMs' prompts, either alongside the video or replacing it entirely. Specifically, there is no component trained or fine-tuned in our approach. On VSI-Bench, augmenting 2B and 4B open models with the spatial code improves average accuracy by +4.1 points over the frames-only baseline, with the largest gains on numeric estimation tasks such as absolute distance (+24.1 points). The results suggest that explicitly computed geometry, delivered through the language channel, recovers spatial competence that small VLMs cannot extract from pixels alone.