Image to 3D models improve shape accuracy using partial test time data
Guiding Image-to-3D Generation with Test-Time Partial Observations
Computer Vision and Pattern Recognition
Summary
Creating 3D models from a single 2D image is tricky because the shape details are often uncertain. The authors found a way to use extra 3D clues available during testing to make the models more accurate, without changing the original model. They do this by comparing what the model predicts with these partial 3D clues using a math approach that checks if rays hit the expected object or empty space. This makes the 3D results better in both shape accuracy and visual quality.
What this means in practice
- •For 3d artists and content creators: Improve the accuracy of 3D models generated from single images when partial 3D scans are available at creation time.
- •For robotics engineers: Enhance object shape estimation by integrating onboard partial depth or geometric sensor data with image-based 3D models at run time.
Authors
Jerred Chen, Simon Weber, Ronald Clark
Abstract
Image-to-3D models can generate visually compelling 3D assets from a single RGB image, but their geometry is often only loosely constrained by the available observations, limiting their use in applications that require geometric fidelity. In many real-world settings, however, partial geometric observations of the object may be available at test time. We introduce a training-free framework for incorporating such evidence into pretrained image-to-3D generative models without retraining or finetuning. To do this, we guide generation using a ray-consistent observation likelihood defined over the model's occupancy representation, combining surface occupancy and free-space evidence. Applied to SAM 3D and its multi-view extension, our approach substantially improves geometric fidelity across different levels of observability, as well as visual quality. Our results demonstrate that pretrained image-to-3D models can effectively integrate partial geometric observations through explicit test-time guidance, complementing their learned generative priors without modifying the underlying model.