LoCoSplat speeds up 3D scene reconstruction with simpler local context use
LoCoSplat: Real-Time Feed-Forward 3D Gaussian Splatting with Minimal 3D Reasoning
Computer Vision and Pattern RecognitionGraphics
Summary
3D reconstruction methods often rely on complex networks to combine multiple camera views into a detailed 3D scene. The authors noticed that the key information needed to adjust simple 3D primitives called Gaussians comes mostly from local surroundings, not from heavy 3D networks. They introduced LoCoSplat, which efficiently uses a simple average of nearby points and a small neural network to improve speed and quality. This method works faster and better on several datasets, using less memory and training time.
What this means in practice
- •For 3d graphics developers: Accelerate multi-view 3D scene reconstruction in real time with reduced computational cost and improved visual quality using simplified local context processing.
- •For augmented reality app developers: Enable faster 3D environment reconstruction from multiple views with lower memory demands for smoother user experiences on GPU-equipped devices.
Authors
Sinan Wang, Jinjin He, Yuchen Sun, Duowen Chen, Shenyifan Lu, Bo Zhu
Abstract
Feed-forward 3D Gaussian Splatting (3DGS) increasingly aggregates multi-view evidence with heavy learned 3D networks. We propose LoCoSplat (Local-Context Splatting), motivated by the observation that a Gaussian is a local primitive: once depth is predicted, what the 3D stage must add (scale, rotation, opacity) depends on the point cloud around each anchor, and a fixed local average of that neighbourhood is enough to supply it, no heavy network required. LoCoSplat realises exactly this average: it splats a 16-d linear projection of the point features into a fine and a coarse grid and reads both back at each anchor with a 0.14M-parameter pointwise MLP; with no learned 3D network and no dynamic sparse computation, its whole encoder runs as one fp16 CUDA graph. On RealEstate10K, LoCoSplat outperforms every prior feed-forward method on PSNR, SSIM, and LPIPS at 6, 12, and 24 views, with a margin that widens as views densify (+3.3 PSNR over VolSplat, the prior voxel-aligned state of the art, at 24 views) and grows further under zero-shot transfer to ACID and fine-tuning on ScanNet. It reconstructs a 6-view scene in 33 ms on one NVIDIA RTX PRO 6000 GPU, the fastest of seven feed-forward methods and $4.2\times$ faster than the previous state of the art, trains $2.7\times$ faster ($5.9\times$ at 24 views), and uses $6.7\times$ less inference memory.