Spatialasm enhances large vision models with active viewpoint thinking
SpatialASM: Internalizing Active Mentalization for Spatial Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Humans often solve puzzles that involve understanding spaces by imagining looking at scenes from different angles. The authors found that current large vision-language models don’t naturally do this kind of active viewpoint thinking. They created SpatialASM, a method that trains these models to simulate changing viewpoints internally, helping them mentally reconstruct scenes better. This improves model accuracy on tasks that require understanding space from multiple angles without losing their general vision skills.
What this means in practice
- •For robotic system designers: Use SpatialASM to enable robots to mentally simulate and adapt their viewpoints for better spatial navigation and object manipulation.
- •For augmented reality developers: Build AR applications that improve interpretation of complex spatial scenes by integrating viewpoint-conditioned mental simulation.$Commercial implications: Allows AR products to deliver richer 3D understanding by modeling viewpoint shifts, enhancing user experience in spatially complex environments.
Authors
Yuhong Liu, Shenxi Wu, Junyi Chi, Beichen Zhang, Xuanlang Dai, Jiaqi Wang, Yuhang Zang, Dahua Lin
Abstract
When solving complex spatial reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints. But can Large Vision-Language Models (LVLMs) proactively think in the same way? We term this ability Active Spatial Mentalization (ASM), an intrinsic component of visual intelligence that remains substantially underdeveloped in current LVLMs. To bridge this gap, we introduce SpatialASM, a scalable framework to internalize ASM into LVLMs through training-time spatial simulation. SpatialASM employs an allocentric novel-view generator to ground hypothetical viewpoints in visual evidence and produces two complementary forms of supervision: i) explicit ASM trajectories that teach LVLMs when and how to adaptively invoke view planning and imagination during spatial reasoning, and ii) automatically-constructed perspective-taking tasks that further strengthens viewpoint-conditioned scene imagination. On seven benchmarks spanning relatively perception-oriented spatial understanding to more challenging tasks requiring multi-image, cross-view and viewpoint-transformation reasoning, SpatialASM yields performance gain on LVLMs of four LVLM backbones in spatial reasoning while retaining their original general vision capabilities. The experimental results demonstrate that our approach improves average accuracy across the seven spatial benchmarks. And the synthesis of ASM trajectories provides a scalable pathway to alleviate the scarcity of spatial Chain-of-Thought (CoT) supervision for eliciting active viewpoint exploration and mental scene simulation.