SpatialScout tests vision models for exploring and mapping 3D rooms
SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?
Computer Vision and Pattern RecognitionComputation and Language
Summary
Understanding a 3D space means not only seeing what’s around but also building a detailed mental map. The authors introduced SpatialScout, a way to test if computer models can explore a room and accurately find objects while moving around, like a robot might do. They found that even the best models now miss many details and only roughly locate objects. Their benchmark helps show where current technology falls short and what needs to improve for better 3D scene mapping.
What this means in practice
- •For robotics engineers: Develop autonomous robot navigation systems that can actively explore indoor environments and identify object locations accurately.
- •For augmented reality developers: Improve AR applications by integrating more accurate 3D maps of indoor spaces generated from active object-level exploration.
Authors
Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li
Abstract
Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.