Fast and efficient rgb only 3d reconstruction with tiny transformer
OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute
Computer Vision and Pattern Recognition
Summary
Creating 3D models from regular photos is usually slow and requires powerful computers or special sensors. The authors introduce OTT3R, a method that uses a small AI model to quickly and efficiently build 3D reconstructions using only color images, running on just two standard GPUs. It also replaces the slow traditional process of preparing training data with a fast, reliable system that works on many image sets where older methods fail. OTT3R can generate accurate 3D information much faster and with less computing power than previous models, making it easier to use in everyday settings.
What this means in practice
- •For computer vision engineers: Build fast and accurate 3D models from RGB images on limited hardware, enabling development of real-time vision applications.
- •For film and game developers: Generate large 3D datasets automatically from images to speed up content creation and environment reconstruction processes.
Authors
Brandon Leblanc, Charalambos Poullis
Abstract
Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks a COLMAP-like system for neural 3D pseudo-label generation. We present OTT3R (RGB-Only Tiny Transformer for 3D Reconstruction), a knowledge distillation framework that addresses both problems on a single workstation equipped with 2 GPUs. Distilling $π^3$ (959M parameters) into a 102M-parameter student yields 9.4$\times$ compression and up to 7$\times$ faster inference, trained at 1.6% of VGGT's training compute. An integrated pseudo-label pipeline offers a reliable, high-throughput alternative to COLMAP, generating dense per-pixel point maps and SE(3) camera poses for a 667K-image corpus in 3.5 hours on two commodity GPUs and succeeding on every sequence we tested, including those where COLMAP fails. The general student tracks the teacher on in-distribution monocular depth and, zero-shot, outperforms COLMAP on 7-Scenes and on DTU completion, but it does not replace the teacher on out-of-distribution multi-view geometry. The deployable artifact is the domain-specialized student: after specialization at 0.2% compute, it is 4$\times$ more accurate than COLMAP on 7-Scenes at 980$\times$ throughput, with near-teacher completion. Code is available at https://github.com/TheFourthKaramazov/OTT3R