Scene reconstruction system creates physics-ready 3d scenes from images
SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Building accurate 3D scenes for robots to use is hard, especially from just one picture. The paper’s authors made SceneRig, a system that not only recreates what objects look like but also where they belong and how they stand physically in a scene. SceneRig uses a process that edits and tests the 3D scene repeatedly in a physics simulator until everything is stable. This makes the reconstructed scenes better for robots to interact with and predict outcomes in the real world.
What this means in practice
- •For robotics engineers: Create realistic 3D scenes from single photos that robots can use to test manipulation in simulation more reliably.
- •For augmented reality developers: Generate accurate and physically stable 3D environments from images to enhance interactive AR experiences.
Authors
Rundong Luo, Yunong Liu, Di Cao, Matthew Tancik, Shyamal Buch, Colton Stearns, Wenqi Xian
Abstract
Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to achieve physical stability. Across diverse inputs, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping the reconstructed scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: its scenes predict real-robot policy outcomes more accurately than the baseline method and more reliably support replay of recorded manipulations without trajectory adaptation.