SceneScaffold builds spatial maps for better 3D scene understanding
SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding
Computer Vision and Pattern Recognition
Summary
3D scene understanding models try to describe what’s in a 3D scene by turning it into a list of things. But this approach loses the spatial layout, making it hard to understand how objects relate or where they are. The authors propose SceneScaffold, which first organizes the 3D scene into parts that keep track of objects, their locations, and their relationships before describing it in language. This organized approach helps language models reason more accurately about complex scenes with many relationships or unclear layouts.
What this means in practice
- •For augmented reality developers: Use a structured 3D scene representation to enable better object interaction and spatial reasoning in AR applications.
- •For robotic navigation engineers: Improve a robot’s understanding of complex environments by providing spatially organized scene information before natural language reasoning.
Authors
Xiangqi Li, Libo Huang, Jiarui Zhao, Weilun Feng, Chuanguang Yang, Zhulin An, Yongjun Xu
Abstract
Recent 3D large multimodal models (3D-LMMs) rely on a visual bottleneck to compress complex 3D scene evidence into a limited number of visual tokens compatible with large language models (LLMs). Current visual bottlenecks, however, often passively compress heterogeneous 3D evidence into a homogeneous object-centric token sequence, leaving the spatial organization of the scene under-represented. This under-representation forces the LLM to recover spatial relations from a flattened token sequence, leading to unstable reasoning in relation-intensive and spatially ambiguous scenes. To address this issue, we propose SceneScaffold, an active scene-state construction framework for unified 3D scene understanding. SceneScaffold reformulates the visual bottleneck from a passive feature compressor into an active scene organizer, constructing a role-aware spatial scaffold before language reasoning. Specifically, SceneScaffold organizes superpoint-level visual evidence into scene-state components with distinct structural roles: entity states preserve core object semantics, scene-frame states maintain spatial references via boundary and region anchors, relation states encode object-environment interaction cues, and a global summary provides compact context. Through this role-aware construction, SceneScaffold provides the LLM with a spatially organized scene representation before language reasoning. Experiments on unified 3D scene understanding tasks, including 3D visual grounding, question answering, and dense captioning, demonstrate the effectiveness of SceneScaffold, while diagnostic results further show its applicability to relation-intensive and spatially ambiguous cases. Code is available at https://github.com/lixiangqi707/SceneScaffold.