Explicit spatial reasoning library improves video model performance
From Reasoning Failures to Composable Video Spatial Intelligence
Computer Vision and Pattern Recognition
Summary
Many vision-language AI models struggle with understanding space and positions in videos because they mix up how to see, measure, or track objects correctly. The authors identified four common errors in how these models handle spatial information. They built a tool called CROSS that works with existing models to fix these errors without extra training. Adding this tool helped various models do better on tests involving spatial reasoning in videos.
What this means in practice
- •For video analytics developers: Integrate CROSS to enhance spatial reasoning and accuracy in video interpretation without retraining existing vision-language models.
- •For robotics engineers: Use the CROSS library’s spatial skills to improve robots’ understanding of object positions and movements in video inputs for better interaction.
Authors
Pengzhan Sun, Junbin Xiao, Ramanathan Rajaraman, Shiu-hong Kao, Angela Yao
Abstract
Spatial reasoning benchmarks evaluate vision-language models across diverse tasks, but task-level scores do not reveal which underlying capabilities account for success or failure. Each task requires recovering spatial evidence, representing geometry, and reasoning over it. We disentangle these capabilities by comparing predicted and ground-truth spatial context under a shared schema and coordinate contract. This comparison reveals four recurring sources of error: inaccurate perception, missing information in the spatial context, selection of the wrong measurement, and errors in reference frames or in tracking position and orientation. Guided by this diagnosis, we develop CROSS, a training-free library of typed geometric operators and spatial skills that function over available evidence to support reliable video spatial reasoning. The resulting library supplies verified context to non-coding VLMs or callable skills to a SpatialClaw agent. We evaluate \methodname{} on five benchmarks. \methodname{} raises the average score from 55.9\% to 60.2\% on ReVSI and improves the SpatialClaw result from 62.8\% to 66.3\% on DSI-Bench. These gains demonstrate that explicit handling of spatial conventions can repair systematic reasoning failures without additional training.