GraftRay enables realistic object insertion into Gaussian splatting scenes
GraftRay: Physically Based Mesh Insertion into Gaussian Splatting Scenes via Path Tracing
Computer Vision and Pattern RecognitionGraphics
Summary
Putting new virtual objects into computer-generated scenes so they look real is hard because lighting and shadows must fit the scene perfectly. The authors created GraftRay, a method that makes inserted objects look naturally lit and cast realistic shadows by understanding the scene’s light sources and materials. It works for scenes built using Gaussian splatting and uses advanced rendering to produce consistent views from different angles. This leads to better quality and more lifelike scenes than previous approaches.
What this means in practice
- •For augmented reality developers: Insert virtual 3D objects into indoor scenes with correct lighting and shadows for AR experiences.
- •For content creation teams: Add realistic virtual items into 3D scene reconstructions for films or video game assets with consistent lighting across views.
Authors
Jinxin Ai, Matthias Niessner
Abstract
Inserting virtual objects into scenes reconstructed via 2D or 3D Gaussian Splatting (2DGS/3DGS) is a key capability for content creation, AR, and synthetic data generation. However, realistic insertion requires the object to be seamlessly lit by the scene and, conversely, to cast physically plausible shadows onto it. Existing approaches either harmonize appearance in image space, sacrificing multi-view consistency, or approximate scene illumination with an environment map at the insertion point, which cannot represent near-field area lights such as windows and lamps that dominate indoor lighting. To address this, we present GraftRay, a unified method for physically consistent insertion of mesh objects into indoor scenes represented by either 2D or 3D Gaussians. We recover scene materials and surface normals with diffusion-based priors, and explicitly model spatially-extended emitters, solving their radiance in closed form. A hybrid renderer then jointly path-traces the scene Gaussians and inserted meshes, producing view-consistent shading, interreflections, and soft shadows. On FIPT, GraftRay sets a new state of the art, outperforming recent baselines such as ComGS and EAG-PT in rendering fidelity and shadow accuracy, while maintaining highly competitive rendering efficiency.