Solv3R transformer updates 3D maps from multiple camera views
Solv3R: 3D Looped Transformer as a Multi-view Solver
Computer Vision and Pattern Recognition
Summary
Building 3D maps from many pictures usually involves adjusting camera positions and shapes step-by-step. Most AI tools try to get the whole 3D shape at once or update slowly as each new picture comes in, making it hard to improve or add later. The authors created Solv3R, a special kind of AI called a looped transformer, that can keep improving a 3D map no matter how many updates it has had, and can add new pictures easily. This method works well for making better 3D reconstructions both offline and in live situations without needing to train separately for each.
What this means in practice
- •For augmented reality developers: Update and refine 3D scenes dynamically as new device views are captured to improve AR experiences without restarting the mapping process.$Commercial implications: Supports live 3D scene updates crucial for commercial AR apps needing smooth, continuous environment scanning.
- •For robotics engineers: Incrementally build and improve robot environment maps from multiple camera inputs during exploration without recomputing prior data.
Authors
Jeong-gi Kwak, Andrea Tagliasacchi, Sho Kagami, Yuki Ono, Kwang Moo Yi
Abstract
Classical structure-from-motion reconstruction proceeds iteratively, refining camera poses and scene geometry from a current estimate. This allows maps to easily expand without recomputing already computed views, or geometry estimates to improve as more observations are provided. Most geometric foundation models, however, predict entire geometry in a single forward pass or update a recurrent state as frames arrive sequentially, making this expansion and refinement non-trivial. We introduce Solv3R, a looped transformer that learns to continue solving multi-view geometry from its current state. Importantly, its update operator is independent of the iteration count, allowing previously refined and newly introduced views to be jointly updated regardless of how ``solved'' they are. We train the operator to preserve accurate reconstructions under continued iteration and correct inaccurate states, enabling refinement of fixed image collections and integration of new observations. Representations of solved views can also be cached once and reused as geometric context across iterations, enabling compact and efficient key-value caching. We analyze the solver dynamics and compare it with the state-of-the-art, demonstrating that a single learned solver supports offline reconstruction, incremental reconstruction, relocalization, and streaming without task-specific training or modules. The code and models will be made publicly available at https://ubc-vision.github.io/solv3r/.