Large vision language models improve 3D spatial reasoning with synthetic blocks
SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how objects in a 2D image fit together in 3D space is hard for big AI models that see and read images. The authors made a simple digital game with colorful blocks stacked in different ways to teach these models to think better about 3D shapes. When the models practiced on this game, they got much better at solving real-world 3D puzzles in photos. This approach is cheaper and less noisy to set up than using real photos with complex 3D labels.
What this means in practice
- •For robotics engineers: Train robots to better understand 3D structure from 2D cameras for grasping and manipulation using synthetic block-stacking tasks.
- •For virtual reality developers: Improve spatial reasoning of AI agents in VR environments by incorporating synthetic datasets focused on 3D-to-2D spatial tasks.
Authors
Soohyun Ryu, Sohee Kim, Eunho Yang
Abstract
Large Vision-Language Models (LVLMs) have achieved strong performance on diverse visual tasks, yet their ability to reconstruct and reason about the 3D structure of the scene depicted in 2D images -- referred to as spatial intelligence -- remains limited. Existing approaches attempt to address this gap by using real-scene spatial question answering datasets that require dense geometric annotations. However, constructing such labels is costly, time-consuming, and often noisy due to reliance on external perception modules. In this work, we propose a novel paradigm inspired by human cognitive development: learning foundational spatial skills through structured block-manipulation tasks. We introduce SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering 3D-to-2D projection, viewpoint transformation, and structural combination. The dataset further incorporates controlled color modulation as visual cues to encourage anchor-based reasoning in visually complex conditions. Experiments demonstrate that LVLMs trained on our dataset through either direct answering or reasoning-based prediction significantly outperform baselines and generalize to real-world spatial tasks, despite the dataset's synthetic and compact nature. Code and data are available at https://github.com/rsoohyun/SpatialBlock.