Better 3D reconstruction from diverse found photos of objects
Trainspotting: Learning Object-Centric 3D Reconstruction from Found Photos
Computer Vision and Pattern Recognition
Summary
Photos of the same object taken by many people over years and places are very different and harder for current AI models to interpret for 3D reconstructions. The authors created a large dataset from Wikimedia Commons with thousands of objects and hundreds of thousands of photos taken in varied environments. They tested existing 3D reconstruction models on this dataset and found they struggled especially when objects moved or were represented by many physical copies. Fine-tuning one model on this dataset improved performance significantly, including on similar independent data.
What this means in practice
- •For computer vision engineers: Improve camera pose and depth estimation models for reconstructing objects from widely varying real-world photos.
- •For digital heritage specialists: Create 3D models of cultural artifacts using found photos from public archives without controlled photo sessions.
Authors
Hanyu Chen, Noah Snavely
Abstract
State-of-the-art feedforward reconstruction models perform poorly on in-the-wild object-centric photo collections. They are trained largely on single-session object captures whose images vary in viewpoint while the object and its environment remain nearly fixed. Public photo archives contain a different form of multi-view data. They hold found photos of the same object or design: photos taken independently by many people across years, environments, cameras, and sometimes physical units. In this work, we introduce Trainspotting, a dataset of 12,068 objects and 311,886 photos from Wikimedia Commons. We reconstruct each object to recover camera poses, multi-view stereo depth, and object masks. We categorize the objects into three regimes: objects fixed at one site, objects that move between sites, and designs shared by many physical units. Evaluating recent feedforward reconstruction models on Trainspotting shows that they fail most severely in the last two regimes. Fine-tuning VGGT-$Ω$ on Trainspotting significantly improves camera pose, depth, and 3D geometry on held-out objects. We observe similar improvements on the independent NAVI in-the-wild benchmark, which contains photos of each object in different environments. Performance on conventional benchmarks remains largely unchanged.