Spatial reasoning gap revealed in multimodal AI with GeoSpatialBench
Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
Artificial Intelligence
Summary
Many AI systems struggle to fully understand and change spatial relationships in maps and city layouts. The authors created GeoSpatialBench, a test that checks how well AI models handle real-world geographic data and maps with precise rules. They found AI models do better when using structured map data but often fail when asked to rearrange places while following strict rules. This shows that recognizing spatial layouts is easier for AI than actively editing them. Their test helps find and improve AI's weaknesses in understanding and handling spatial information.
What this means in practice
- •For urban planning teams: Test AI tools for modifying city layouts that respect real-world spatial rules using structured geographic data.
- •For mapping software developers: Improve map editing features by benchmarking AI's ability to adjust spatial configurations under constraints.
Authors
Zheng Ren, Yanning Wang, Ziheng Huang, Fei Huang, Shiqi Wang, Ziqian Ye, Yang Yue
Abstract
Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.