Complete 3D scenes and separate objects from one RGB-D image
CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots often have trouble understanding cluttered rooms by only seeing part of what’s there. The authors created CODA, a system that first recreates the whole scene from a single RGB-D image and then picks apart the objects within it. It uses special techniques to make sure the recreated scene matches the part of the room the robot actually saw, fixing common problems where objects are missed or merged together. Tests showed CODA produces more accurate scene models and keeps objects in stable positions compared to other methods.
What this means in practice
- •For robotics teams: Improve robot understanding of cluttered indoor spaces with more accurate and stable 3D scene reconstructions from single RGB-D images.
- •For augmented reality developers: Generate complete and accurately segmented 3D environments from partial RGB-D inputs to enhance mixed reality experiences indoors.
Authors
Dongwon Son, Junhyek Han, Yoontae Cho, Minseok Lee, Hong-seok Choi, Jiwook Choi, Hyungjin Kim, Beomjoon Kim
Abstract
Robots operating safely in cluttered everyday environments often need to infer scene geometry from partial observations. Methods that detect objects in 2D and reconstruct them independently struggle in such scenes: a missed object is never reconstructed, a merged detection can fuse two objects, and separately reconstructed meshes may overlap or fail to touch their supporting surfaces. We introduce CODA (Complete Once, Decompose Afterward), a generative model that instead reconstructs the complete scene geometry from a single unsegmented RGB-D image, then separates the surface into the surrounding environment and movable objects. Still, generated scene geometry can drift from the observed partial point cloud. To reduce this drift, CODA uses two explicit 3D grounding mechanisms to keep reconstructed geometry consistent with observed surfaces while completing unseen regions. Experiments on HomebrewedDB and our custom cluttered-scene dataset show more accurate reconstructions and a higher fraction of objects remaining in place under simulated gravity than both object-first and scene-first baselines.