Render dedicated geometry improves novel view synthesis quality
RDGSplat: Render-Dedicated Geometry for Novel View Synthesis
Computer Vision and Pattern Recognition
Summary
Creating new images of an object from different viewpoints is hard because the 3D shapes used often do not match how the views actually look. The authors propose a way to create a second 3D shape specifically for making better images, without changing the original shape used for measurements. By keeping the first shape unchanged and focusing on this second one, their method improves the quality of new views made from fixed pretrained systems. This lets the original 3D information stay accurate while the rendered images look better.
What this means in practice
- •For augmented reality developers: Improve virtual object rendering quality in AR apps without retraining existing 3D models, enabling more realistic scene integration.
- •For autonomous vehicle software teams: Generate better novel views of environments from fixed depth models to improve simulation and perception without changing core sensors.
Authors
Zhijie Zheng, Xinhao Xiang, Jiawei Zhang
Abstract
3D foundation models enable efficient novel view synthesis by carrying a Gaussian head on the representation they already use for reconstruction. However, the views they render fall short of the geometry they recover, because that geometry is estimated under a metric objective and never scored on how it renders. Recent methods alleviate this by updating the backbone weights, but they thereby discard the metric predictions the model was built for and must be repeated for every new backbone. To this end, we propose RDGSplat, a framework that decodes a second geometry dedicated to rendering from a frozen 3D foundation model, leaving its metric predictions intact. In particular, we devise Render-Dedicated Geometry Decoding, which duplicates the pretrained decoders and optimizes the duplicates under photometric supervision alone. Then, a Target-Pose Conditioned Adapter is introduced to reformulate the representation those decoders read, conditioned on the target camera pose rather than the target image. Extensive experiments show that RDGSplat improves novel view synthesis across three feed-forward backbones on four benchmarks, with every pretrained weight frozen. On RE10K, it raises WM2.0 from 20.918 to 24.266\,dB while training 205.5\,M added parameters against a frozen 1.4\,B backbone, and the depth and pose the same model predicts are unchanged.