Foundation model method reduces drift in long 3D reconstructions
CLoSeR: Closing the Loop for Long-Context Streaming Reconstruction
Computer Vision and Pattern RecognitionRobotics
Summary
3D reconstruction models that build large scenes over time often get less accurate because small errors add up. The authors show a way to detect when the model revisits places (loops) and use that to correct the position errors. They keep the scale consistent across the whole scene and optimize the frame positions with fewer mathematical complications. Their approach works well on very long sequences, making large-scale 3D reconstructions more accurate and stable.
What this means in practice
- •For mapping and surveying teams: Create accurate 3D maps of large outdoor environments at kilometer scales without accumulated errors distorting the result.
- •For robotics engineering teams: Improve autonomous robot localization and navigation by reducing drift in streaming 3D scene reconstruction over long distances.
Authors
Moyang Li, Zihan Zhu, Wei Zhang, Marc Pollefeys, Daniel Barath
Abstract
Feedforward foundation models have recently shown remarkable 3D reconstruction capabilities. However, existing models exhibit large tracking drift in long-context streaming reconstruction due to error accumulation. In this paper, we revisit loop closure with streaming reconstruction foundation models to enable accurate, drift-free, kilometer-scale reconstruction. Specifically, our method detects loop candidates through global descriptor retrieval, and constructs loop-conditioned windows to estimate the relative poses between looped frames. Given the observation that our adopted streaming reconstruction backbone produces a globally consistent scale, we optimize all frame poses on the SE(3) manifold with sequential and loop closure constraints, avoiding the pose graph optimization on the Sim(3) or higher-dimensional SL(4) manifolds employed in prior works. Extensive experiments show that our method reduces drift and produces consistent geometry on kilometer-scale sequences, significantly outperforming the state of the art. Code is available at https://github.com/MoyangLi00/CLoSeR.git.