Sceneplacer improves placing 3d objects in scenes from partial views
ScenePlacer: Reformulating Scene-Level Object Pose Estimation as Visibility Field Learning
Computer Vision and Pattern Recognition
Summary
Placing 3D objects correctly in a scene is hard when you only have partial views of the scene. The authors created ScenePlacer, which separates how the object is generated from how its position and orientation are found. It uses a visibility field to figure out which parts of the object should be seen from the camera view, helping it align the object accurately in the scene. ScenePlacer does this without needing full scene information during training and works better than other methods on a standard 3D dataset.
What this means in practice
- •For 3d graphics developers: Place generated 3D objects into virtual scenes accurately using only partial depth data without retraining generative models.
- •For augmented reality developers: Improve alignment of virtual objects with real-world scenes when only partial 3D observations are available from cameras or depth sensors.
Authors
Ken Deng, Jiarui Liu, Guibiao Liao, Kunming Luo, Mochu Xiang, Hongyu Yan, Yebin Liu, Shenghua Gao, Yizhou Yu, Ping Tan
Abstract
We present ScenePlacer, a framework for scene-level pose estimation of objects generated by state-of-the-art native shape generation models, placing generated 3D objects into scenes given a partial geometric observation without modifying the underlying generative pipeline. Existing scene generation approaches either generate objects directly in scene coordinates, compromising their canonical-space priors, or regress poses from complete generated shapes in the shape generation process - the latter being a poorly conditioned problem that is sensitive to scale ambiguity and partial observations. ScenePlacer decouples object generation from pose estimation by reformulating alignment as conditional visibility-field learning. Given an observed partial point cloud from reconstruction and a generated mesh, it predicts a visibility field defined in object space, then estimates the rigid transformation by aligning the partial observation to visible parts on the mesh. We propose a pretrained 3D shape VAE as our backbone, preserving its mesh prior while conditioning it on a separately encoded observation. We further introduce visibility-aware, radius-constrained supervision and synthesize training pairs using only object-level meshes and rendered views, eliminating the need for scene-level pose annotations. On 3D-FUTURE, ScenePlacer consistently outperforms existing placement and pose-refinement baselines under both ground-truth and monocular depth estimation. With generated objects, it achieves a Chamfer distance of 0.0077 and an F-score of 92.40 under estimated depth and improves to 0.0023 and 98.27, respectively, under ground-truth geometry.