Improving real-time 3D tracking with dual covariance Gaussian splatting
Dual Covariance Gaussian Splatting SLAM: Decoupling Rendering and Registration for Robust Real-Time Tracking
RoboticsComputer Vision and Pattern Recognition
Summary
Tracking 3D scenes in real time involves matching new camera views to a map of the environment. The authors found that using one kind of shape uncertainty (covariance) for both making images and aligning frames causes problems. They propose giving each 3D point two kinds of uncertainty: one optimized for making clear images and another based on sensor noise for better alignment. This approach improves tracking accuracy and speed across indoor and outdoor scenes.
What this means in practice
- •For roboticists: Implement SLAM systems that achieve more accurate and robust real-time localization by using dual covariances for rendering and registration.
- •For augmented reality developers: Enhance AR experiences on RGB-D devices by reducing tracking errors and maintaining real-time frame rates with dual covariance Gaussian models.
Authors
Edward Beng Wai Tan, Siew-Kei Lam
Abstract
ICP-based 3D Gaussian Splatting (3DGS) SLAM tracks in real time by registering incoming frames against map Gaussians, using each primitive's covariance for both rendering and registration. These two uses place conflicting demands on one covariance. The mapper shapes it to minimize photometric error, often flattening it against surfaces, while robust registration typically benefits from measurement uncertainty. We propose a dual-covariance parameterization. Each Gaussian keeps a single mean but holds two covariances: a rendering covariance optimized by the mapper, and a tracking covariance derived from an RGB-D sensor noise model. We further use the tracking covariances as Gaussian anchors for image corners, providing constraints in directions where depth geometry is weak. We evaluate on TUM RGB-D, ScanNet, Replica, and two outdoor sequences recorded with a RealSense D435i on wheeled and handheld platforms. We achieve robust tracking performance across multiple scenes and reduced odometry drift, while tracking at $\sim$ 60 FPS.