LiteReality-Agent reconstructs detailed 3D indoor scenes from RGB-D scans
LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction
Computer Vision and Pattern RecognitionGraphicsRobotics
Summary
Building detailed 3D models of indoor spaces is hard because it requires combining lots of information precisely. The authors present LiteReality-Agent, a system that acts like a smart helper using special tools to gather evidence and improve its 3D model step-by-step. It writes and revises a Python script that can recreate the room accurately, making the model ready for simulations or AI experiments. The system also has checks to make sure the model is correct and realistic, performing better than earlier approaches.
What this means in practice
- •For robotics developers: Generate accurate and simulation-ready 3D room models from RGB-D sensor data to test embodied AI navigation and interaction algorithms.
- •For game environment designers: Create realistic indoor scenes as Python-executable scripts that can be easily edited and integrated into simulation engines for interactive content development.
Authors
Zhening Huang, Yueyan Li, Johnathan Chiu, Xiaoyang Lyu, Matt Zhou, Yuxin Yao, Joan Lasenby, Shangzhe Wu
Abstract
We present LiteReality-Agent, an agentic system for reconstructing real indoor environments as realistic, articulated, and simulation-ready 3D scenes from RGB-D scans. At its core, LiteReality-Agent formulates 3D reconstruction as a coding problem, in which a coding agent gathers evidence using specialised tools and iteratively edits a Python script, Room.py, which can be executed to produce a 3D digital twin of the room. With this formulation, we develop a robust observe-edit-verify harness that supports evidence gathering, measurement, verification, layout optimisation, simulation readiness, and quality control throughout the reconstruction process. LiteReality-Agent produces high-quality reconstructions suitable for simulation and downstream embodied AI tasks. Furthermore, as agent capabilities continue to improve rapidly, the system introduced by LiteReality-Agent remains a strong orchestration framework for future agents: it equips them with specialised tools, structured workflows, and robust verification mechanisms that substantially improve reconstruction quality and reliability. We demonstrate that LiteReality-Agent produces reconstructions that are more geometrically accurate, visually realistic, and simulation-compatible than those generated by recent frontier models, such as Astra and Fable. We therefore view LiteReality-Agent as a practical and important building block for robust real-to-sim systems. Both the source code and the data-capture application are publicly available. Code:https://github.com/LiteReality/LiteReality-Agent/