Sparse radiance voxels enable detailed scalable 3D maps from robot data
SpaRV: Sparse Radiance Voxels for Scalable Geometrically Consistent LiDAR-Visual Mapping and Rendering
RoboticsComputer Vision and Pattern Recognition
Summary
Making realistic and accurate 3D maps from robot images and scans is hard because cameras don’t always capture every angle, and small errors blur the results. The authors created SpaRV, which combines sparse 3D points from LiDAR with images to build clearer and more consistent maps even far from where the robot moved. SpaRV also fixes small camera mistakes during map building and manages memory use to handle large areas. This method works well enough to reconstruct a whole university campus using thousands of images on a single GPU.
What this means in practice
- •For robotics engineers: Generate accurate and photorealistic 3D maps from sparse robot sensor data with improved geometry and texture quality.
- •For simulation developers: Create scalable and detailed virtual environments from large-scale real-world scenes captured by robots for realistic simulation.
Authors
Jianheng Liu, Yunfei Wan, Fangming Cheng, Xiangyu Hong, Zuhao Zou, Haotian Li, Longji Yin, Chunran Zheng, Jiarong Lin, Fu Zhang
Abstract
Robotic simulation, navigation, and inspection need photorealistic and geometrically accurate maps. Radiance fields achieve photorealism, but building them from robot data raises three challenges. A robot's trajectory serves its task, not reconstruction, so images from few viewpoints leave the geometry under-constrained. Residual errors in camera poses and intrinsics blur the map. The map's GPU memory grows with scene size, which limits its level of detail in large scenes. We present \textbf{SpaRV}, a \textbf{Spa}rse \textbf{R}adiance \textbf{V}oxel map rendered by differentiable ray marching, which lets irregular LiDAR scans supervise it alongside images. LiDAR depth and occupancy anchor the under-constrained geometry, and normal and smoothness regularizers keep it smooth between LiDAR points. To correct camera errors, SpaRV alternates map training with photometric optimization of the camera poses and focal lengths. To bound GPU memory in training and rendering, SpaRV holds one submap at a time at full resolution and the rest at coarser octree levels. Experiments show that SpaRV keeps the geometry consistent away from the captured trajectory and sharpens textures by optimizing the camera parameters. On one 24\,GB GPU, SpaRV reconstructs a university campus captured in 63{,}582 images and simulates other cameras and LiDARs. Our code will be released at \url{https://github.com/hku-mars/SpaRV}.