SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding
2026-08-05 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed SmartMage, a model that helps computers better understand 3D scenes by smartly choosing which types of information to use depending on the question asked. Unlike older models that use all available information no matter what, SmartMage selects the most relevant visual and geometric data to avoid confusion and save computing power. It uses special modules to pick and combine information guided by the meaning of the task. Tests showed SmartMage performs very well on several 3D scene tasks and also does well on video understanding. The authors also created a benchmark to study how the model decides which information to use for different categories.
3D scene understandingmultimodal learninglarge language modelsvisual modalitiesgeometric cuesmodality selectionsemantic guidanceadaptive routingmultimodal reasoningbenchmarking
Authors
Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun, Hehe Fan
Abstract
Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.