VoxelTTO improves 3D scene reconstruction and camera pose accuracy
VoxelTTO: Voxel-Aligned Feed-Forward 3D Gaussian Splatting with Test-Time Optimization
Computer Vision and Pattern Recognition
Summary
Creating accurate 3D images from many pictures can be tricky because overlapping parts and camera mistakes cause visual errors. The authors developed VoxelTTO, a method that builds 3D scenes by grouping image data into cubes (voxels) instead of tying features directly to pixels, which reduces errors. It also fine-tunes some parts during testing to fix camera angle mistakes without changing the whole system. Their approach makes the 3D images clearer and camera positions more precise compared to older methods.
What this means in practice
- •For computer vision engineers: Build more accurate 3D models from multiple images improving novel view synthesis and camera pose estimation for augmented reality and robotics.
- •For 3d content creators: Generate better aligned and visually faithful 3D scenes with less artifact and overlap compared to traditional pixel-aligned methods.
Authors
Yibin Zhao, Yihan Pan, Yangwen Li, Jun Nan, Jianjun Yi
Abstract
Recent feed-forward 3D Gaussian Splatting (3DGS) methods typically regress pixel-aligned Gaussian primitives, often causing excessive overlap and artifacts, while inaccuracies in predicted camera poses can lead to misalignment in novel-view synthesis (NVS). We present VoxelTTO, a feed-forward framework for reconstructing geometrically accurate 3DGS scenes from an arbitrary number of images and optional camera parameters. VoxelTTO aggregates dense image features into a global voxel representation and decodes Gaussians from voxel features, breaking the pixel-to-Gaussian correspondence. To exploit known camera parameters while keeping the pretrained visual foundation model (VFM) parameters frozen, we introduce test-time optimization (TTO) that adapts lightweight LoRA modules using pose supervision. We further replace vanilla 3DGS rasterization with stochastic solid volume rendering during training and inference, improving geometric fidelity. Training updates only the voxel-aligned Gaussian reconstruction modules, requiring 80 GPU hours. Experiments on Replica, Tanks and Temples, and DTU demonstrate improved RGB-D NVS and camera-pose estimation relative to prior methods.