Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed GAMSI, a new model that understands 3D space and size from just regular RGB images, without needing extra data like depth maps or point clouds. Their approach uses two sets of special queries to separately focus on detailed size information and overall 3D structure, while keeping these two types of information from mixing. They also include a module to refine these cues by comparing them to established visual models during training, but the model doesn't rely on those models when running. By training on a large combined dataset with many spatial tasks, their model performs very well on several spatial understanding tests.
3D structural perceptionmetric scale estimationmultimodal large language modelsRGB imagesMetric-Structure Decoupled QueriesExpert-Guided Visual Groundingspatial instruction-tuningautoregressive backbonevisual groundingspatial intelligence benchmarks
Authors
Yufei Zheng, Xuhan Zhu, Zide Liu, Chunpeng Zhou, Chenfeng Wang, Yongchao Xu, Yunnan Wang, Jiawei Liu, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha
Abstract
Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152{,}776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks.