Feed-forward model improves 3D scene capture with faster Gaussian refinement
GradSplat: Gradients Inside the Forward Pass for Feed-Forward 3D Gaussian Splatting
Computer Vision and Pattern Recognition
Summary
Capturing 3D scenes from a few pictures can be tricky, and the usual automatic methods sometimes get shapes wrong. The authors created a way for their model to fix these shape mistakes during its own process, by learning small adjustments that improve how the 3D spots (called Gaussians) fit the images. This new approach makes the model's predictions much closer to those made by slow, specialized scene-by-scene fixes but works faster and with no extra info. Their method performed best in tests with several sets of pictures.
What this means in practice
- •For augmented reality developers: Improve 3D scene reconstructions from limited input images for AR experiences using a fast, learned gradient refinement inside the model inference.
- •For 3d content creators: Generate more accurate 3D representations from a few photos quickly without manual tuning or extra target views.
Authors
Jingong Chen, Linlian Jiang, Guangyuan Li, Xinxin Zuo, Yang Wang, Chulwoo Pack, Kyle Gao, Kwanghee Won
Abstract
Feed-forward 3D Gaussian Splatting predicts a scene from a few images, but our predictor is less accurate than its own Gaussians optimized per scene on those images. This gap lies mostly in the shape of the Gaussians, their scale and rotation, since optimizing the shape alone recovers almost all of it and opacity and color recover less. Gradient steps on the input views correct this shape, and the model holds these views at inference. Log-anisotropy correction changes the axis ratios of each Gaussian in log space at constant volume, and in-forward gradient refinement renders the Gaussians into the input views and applies the gradient of the photometric loss with a learned recurrent update, first to opacity and color and then to the shape. GradSplat is a feed-forward model that refines its own Gaussians by learned gradient steps inside its forward pass, with no target view. On both complete six-view RealEstate10K protocols, it has the best PSNR and SSIM among nine methods and keeps its accuracy from $6$ to $42$ input views. Without fine-tuning, it is the most accurate on DL3DV and has the best PSNR and SSIM on ScanNet++. The code will be released at https://github.com/Jingong-Chen/GradSplat/.