3D language model improves scene understanding with adaptive pruning and 2D features
3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement
Computer Vision and Pattern Recognition
Summary
Understanding 3D scenes with language models can be tricky because they often include lots of unrelated objects that confuse the model, and they usually only use 3D shapes without helpful details like color or texture. The authors created a system called 3D-SceneQ that smartly picks out only the important objects for a given question, and adds extra information from 2D images to better describe each object. This makes the system better at answering questions and describing 3D scenes. Their tests show it works better than other methods on popular benchmarks.
What this means in practice
- •For augmented reality developers: Improve AR applications by enabling more accurate and focused understanding of 3D scenes based on user queries.
- •For robotics software engineers: Enhance robot perception by integrating rich 2D and 3D information for better object recognition and scene reasoning.
Authors
Yongqi Shan, Lu Zhang, Jiazuo Yu, Yunzhi Zhuge, Huchuan Lu
Abstract
Current 3D scene understanding pipelines typically concatenate vast numbers of 3D object tokens with text tokens and feed the resulting sequence to a Large Language Model (LLM). However, existing 3D-LLMs for scene understanding encounter two major limitations: (i) excessive inclusion of task-irrelevant object data, introducing noise that reduces reasoning accuracy and increases hallucinations; and (ii) reliance solely on point cloud data, which inherently lacks the rich semantic information available in complementary 2D modalities, such as color, material properties, texture, and high-level contextual relationships. To address these challenges, we introduce 3D-SceneQ, a 3D LLM that unifies adaptive token pruning with multimodal semantic enrichment, markedly advancing scene understanding, reasoning, and grounding. Specifically, we propose a Query-Guided Adaptive Pruning (QGAP) module that adaptively selects task-relevant objects based on the intended meaning of user instructions, leveraging a learnable latent query to retain critical information while effectively suppressing irrelevant noise. In addition, we introduce a Multi-modal Object-level Feature Enhancement (MOFE) module, which integrates 2D feature embeddings from pre-trained models into 3D representation to enrich object-level semantic information. Evaluations show that 3D-SceneQ outperforms current state-of-the-art methods across multiple benchmarks (e.g., +5.7 CIDEr on Scan2Cap and +17.0 EM@1 on SQA3D), demonstrating its remarkable capabilities in various 3D language reasoning tasks.