Image to 3D models improved with extra shape hints at test time
Guiding Image-to-3D Generation with Test-Time Partial Observations
Computer Vision and Pattern Recognition
Summary
Generating 3D models from a single photo is tricky because the shape details are often unclear. This paper shows how existing 3D generation models can use partial real-world shape information available during use, like some depth points or surface views, without needing to retrain the model. The authors guide the 3D shape creation by checking which parts match the given shape clues, improving accuracy and realism. Their method works by combining evidence about where the object’s surfaces are and where empty space is. This helps the 3D results look better and fit the actual object more closely.
Image-to-3D generationgeometry fidelitypartial observationstest-time guidanceoccupancy representationray consistencysurface occupancyfree space evidencepretrained modelsmodel conditioning
Authors
Jerred Chen, Simon Weber, Ronald Clark
Abstract
Image-to-3D models can generate visually compelling 3D assets from a single RGB image, but their geometry is often only loosely constrained by the available observations, limiting their use in applications that require geometric fidelity. In many real-world settings, however, partial geometric observations of the object may be available at test time. We introduce a training-free framework for incorporating such evidence into pretrained image-to-3D generative models without retraining or finetuning. To do this, we guide generation using a ray-consistent observation likelihood defined over the model's occupancy representation, combining surface occupancy and free-space evidence. Applied to SAM 3D and its multi-view extension, our approach substantially improves geometric fidelity across different levels of observability, as well as visual quality. Our results demonstrate that pretrained image-to-3D models can effectively integrate partial geometric observations through explicit test-time guidance, complementing their learned generative priors without modifying the underlying model.