CadSplat builds photorealistic 3D twins from few photos using CAD shapes
CADSplat: Sparse-View 3D Gaussian Splatting Aided by CAD Models for Robust, Photorealistic Digital-Twin Reconstruction
Computer Vision and Pattern RecognitionGraphics
Summary
Capturing detailed 3D images of objects from only a few photos is usually hard. The authors created a method called CADSplat that uses simple 3D shapes from computer-aided design (CAD) models to help build accurate and realistic 3D representations from less than 15 photos. They match the shape of an object’s outline to a CAD model, then fine-tune the 3D points to fit the real object better. Their method works well even with very few photos and is useful for things like augmented reality and estimating the poses of objects in pictures.
What this means in practice
- •For augmented reality developers: Register virtual objects on real-world items without markers by matching images to CAD-based 3D models from few photos.
- •For industrial simulation teams: Use photorealistic 3D digital twins derived from sparse images for physical simulations and product design validation.
Authors
Kristof Overdulve, Lode Jorissen, Nick Michiels
Abstract
We present CADSplat, a framework that reconstructs photorealistic, geometrically accurate digital twins from sparse ($<15$ views), wide-baseline posed images of an object by regularizing 3D Gaussian Splatting (3DGS) with an explicit CAD shape prior. Using such a prior requires finding a CAD model whose shape resembles the object depicted in the images and determining the pose of each camera relative to the object. We obtain both by matching segmented object silhouettes against silhouettes rendered from a CAD library and keeping the camera-to-object poses of the best-matching model. We then anchor 3D Gaussian primitives to the surface of the retrieved model and jointly optimize the 3DGS parameters, the camera-to-object registration, and a non-rigid deformation field to account for shape differences between the physical object and the CAD model. Across two real-world datasets, CADSplat outperforms unconstrained, few-shot, and mesh-texturing baselines and degrades gracefully to as few as 3 views. Our experiments show that most of the gain in rendering quality comes from how the splats are constrained---a fixed set of splats tied to a surface and moved by a single smooth deformation field---rather than from the CAD shape itself. The CAD model adds shape knowledge where views are scarcest, in the sparsest captures and on strongly self-occluded objects, and it places every camera in the object's own frame. This enables applications beyond novel-view synthesis, such as markerless augmented reality registration, per-image object pose estimation, physical simulations, and the transfer of part labels from the design to the reconstruction.