MAGiSt3R: Multi-Agent Feed-forward 3D Reconstruction from Monocular RGB Videos
Computer Vision and Pattern Recognition
Summary
The authors introduce MAGiSt3R, a system that creates 3D models from single-camera video quickly and accurately. It uses a special process to turn video frames into small 3D maps, then merges these maps to form a complete 3D scene. To keep track of the camera's position and reduce errors over time, they optimize the camera poses. Tests show that MAGiSt3R performs better in reconstructing scenes and tracking camera movement than other existing methods.
Authors
Ziren Gong, Xiaohan Li, Fabio Tosi, Ninghui Xu, Stefano Mattoccia, Jianfei Cai, Matteo Poggi
Abstract
This paper presents MAGiSt3R, a multi-agent 3D reconstruction framework performing reconstruction and camera tracking for monocular RGB videos at almost 10 FPS. MAGiSt3R relies on a feed-forward model from the 3R family to process RGB videos and regress local point maps, and on a merging model, MAGMA, that combines local maps at both intra-agent and inter-agent levels to obtain the final global point map. Furthermore, MAGiSt3R performs pose graph optimization to mitigate cumulative camera drift occurring along the feed-forward pipeline. We evaluate MAGiSt3R on both synthetic and real-world datasets, demonstrating its superior reconstruction and camera tracking accuracy compared to state-of-the-art approaches.