Model tracks and reconstructs 3d scenes from any image set
ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild
Computer Vision and Pattern Recognition
Summary
Capturing moving scenes from different cameras or times is tricky because images vary in viewpoint and timing. The authors present ARROW, a model that can combine any set of images to rebuild 3D scenes and track points moving in 3D space. It learns to handle different input orders and diverse data types, improving its ability to track objects and reconstruct scenes. This approach performs better than existing methods on multiple 3D tracking benchmarks and works well on 3D reconstruction tasks too.
What this means in practice
- •For computer vision teams: Integrate ARROW to track moving objects and reconstruct 3D scenes from mixed camera sources or times without task-specific retraining.
- •For robotics developers: Use ARROW’s 3D point tracking and reconstruction for robots navigating dynamic environments with unpredictable camera inputs.
Authors
Ilya Fradlin, Christian Schmidt, Jens Piekenbrinck, Karim Knaebel, Gonzalo Martin Garcia, Bastian Leibe
Abstract
Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights are publicly available.