Multimodal models improve 3D spatial reasoning with hierarchical data
MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modeling
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how objects relate in three dimensions from many viewpoints is hard for current AI models that combine images and language. The authors created a new dataset called MV-STRIDE that organizes spatial reasoning tasks in a step-by-step way, similar to how humans think about space. They also developed a method to train models using this dataset that helps the models solve 3D reasoning problems more reliably. Their approach outperforms previous methods at tasks requiring understanding scenes from multiple views.
What this means in practice
- •For robotics engineers: Improve robots' ability to understand and reason about 3D environments by training vision-language models with hierarchical spatial tasks.
- •For augmented reality developers: Enhance AR systems' spatial reasoning across multiple viewpoints using models trained on MV-STRIDE to support accurate 3D scene comprehension.
Authors
Jin Xu, Xiaojian Huang, Zhuodong Luo, Zhihong Zhang, Xin Liu, Jiansheng Wei, Xinzhi Wang, Jie Zhao, Xuejin Chen
Abstract
Despite the rapid progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, robust multi-view spatial reasoning remains a fundamental bottleneck due to the lack of structured 3D cognitive pathways in existing datasets. To address this, we introduce MV-STRIDE, a Multi-View hierarchical SpaTial Reasoning dataset with Interdependent and DEcomposed capabilitiEs. Moving beyond flat data structures, MV-STRIDE explicitly models the dependency relationships between foundational perception, scene understanding, and complex contextual reasoning, providing a coherent learning pathway aligned with human spatial cognition. We develop a systematic QA generation pipeline leveraging diverse 3D scene sources that enforces cross-view dependency constraints to prevent single-view solvability, generating multi-level spatial reasoning tasks supported by cognitively grounded chain-of-thought supervision for complex inference. Extensive evaluations demonstrate that our multi-stage training framework based on our hierarchical dataset achieves state-of-the-art performance across multiple spatial reasoning benchmarks, notably the multi-view oriented MMSI-Bench. Our approach enables MLLMs to maintain robust, 3D-consistent spatial reasoning across diverse viewpoints. The code and dataset are available at https://co1dspring.github.io/MV-STRIDE/.