Feed-forward 3D models improve depth and pose without labeled data
PURE: Refining Feed-forward 3D Reconstruction with Unlabeled Post-training
Computer Vision and Pattern Recognition
Summary
3D reconstruction models that use fast, direct methods can struggle to understand videos with moving people, changing light, and objects blocking the view. The authors propose PURE, a way to make these models better after training, without needing any special labels like depth or camera angles. PURE uses clever clues from 2D information and relationships in the video to improve the 3D understanding of scenes. Their tests show that PURE helps many leading models work more reliably on real-world dynamic videos.
What this means in practice
- •For video processing engineers: Enhance 3D reconstruction accuracy in dynamic videos without requiring expensive depth or pose labels.
- •For augmented reality developers: Improve robustness of real-time 3D scene reconstruction under challenging conditions like movement and occlusion.
Authors
Yaolong Zhu, Peng Li, Jiahao Lu, Chengfeng Zhao, Mengfei Li, Chao Wang, Wenyue Chen, Zekai Gu, Chengwei Ren, Ronggang Wang, Yuan Liu
Abstract
Feed-forward 3D reconstruction models have achieved remarkable progress through large-scale training and strong learned priors. However, they remain fragile when applied to dynamic in-the-wild videos, where non-rigid motion, occlusions, and illumination changes are prevalent. A key limitation lies in their reliance on expensive depth and camera pose annotations, which hinders scaling to diverse real-world dynamic data and restricts generalization. To address this issue, we propose \textbf{PURE}, an unlabeled post-training refinement framework that improves depth and camera estimation without requiring ground-truth supervision. Instead of relying on annotations, PURE elegantly distills robust structural knowledge from state-of-the-art 2D priors into 4D feed-forward architectures. PURE leverages relative depth constraints, affine-invariant geometry distillation, and pose refinement based on feature correspondences to stabilize geometric predictions under challenging dynamic conditions. Extensive experiments on both synthetic and real-world dynamic benchmarks demonstrate that PURE delivers consistent improvements across multiple state-of-the-art architectures. Furthermore, our data-efficient paradigm effectively mitigates domain shifts, successfully unlocking the architectural limits of these base models in challenging, in-the-wild dynamic scenes.