Octree residual network enables efficient real-time multi-modal 3D mapping
OREN-X: Octree Residual Network for Real-Time Multi-Modal Mapping
Robotics
Summary
Robots need to understand their surroundings in many ways to do different tasks, like moving safely or recognizing objects by description. This paper presents OREN-X, a method that stores 3D shapes, light details, and language-based information all together in one special data structure called an octree. By combining these different types of information, the method works faster and more accurately than previous approaches that treated each type separately. The authors tested OREN-X and showed it maps environments quickly while improving accuracy and understanding compared to earlier methods.
What this means in practice
- •For robotic system developers: Create real-time environment maps combining geometry, lighting, and language features for improved robot navigation and scene understanding.
- •For augmented reality developers: Render accurate 3D scenes with integrated radiance and semantic info in real time to enhance immersive AR experiences.
Authors
Zhirui Dai, Qihao Qian, Dinh Minh Nguyen, Quan-Dung Pham, Kiana Bronder, Carlos Nieto-Granda, Yiyu Chen, Quan Nguyen, Nikolay Atanasov
Abstract
To achieve general-purpose autonomy over long horizons, a robot needs to maintain spatial environment information that supports a variety of tasks: geometry for planning and control, radiance for rendering and relocalization, and vision-language features for open-vocabulary grounding. Existing methods represent and estimate each modality separately, multiplying memory and compute cost while forgoing potential synergy among the representations. We develop OREN-X, an online mapping method that uses an octree in 3D space as a shared data structure for indexing and storing a multi-modal field, capturing geometric, radiance, and vision-language information. OREN-X provides efficient unified storage and retrieval of these data in explicit/implicit and full/compressed form. Our unified representation yields cross-modality synergy: SDF estimates are sharpened by occupancy and radiance, while GPU-based ray-octree traversal and octree query enable real-time rendering. We also use online dictionary learning to compress the vision-language features, shrinking them 3.7x below full per-vertex storage while raising the query accuracy. On Replica, OREN-X maps in real time (80+ fps for SDF and 30+ fps for all four modalities), improves near-surface SDF accuracy by 33% over single-modality baselines, and improves mean open-vocabulary 3D mIoU by 71% and mean accuracy by 61% over the best prior method.