Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction
2026-09-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors found that current online 3D reconstruction methods struggle with long videos because they rely on poses relative to just the first frame, causing errors to pile up over time. They noticed that the estimated depth per frame stays accurate, meaning the problem is mainly with tracking the global position. To fix this, they created Scal3R, which uses small learnable tokens to relate the current frame's pose to multiple previous keyframes instead of just one. This method, combined with an optimization step to reduce drift, improves accuracy significantly and works well on several benchmark datasets.
3D reconstructionpose regressiondepth estimationpose graph optimizationloop closureonline algorithmskeyframesKITTI datasetasymmetric attentionmulti-reference querying
Authors
Chin-Yang Lin, Yang-Che Sun, Cheng Sun, Fu-En Yang, Min-Hung Chen, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu
Abstract
Online 3D reconstruction models perform poorly on long videos. This happens because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. Small drifts accumulate and amplify into significant geometric collapse. However, we observe that per-frame depth remains stable throughout this failure. The backbone's local geometry remains intact; only the global pose head breaks down. Motivated by this decoupling, we introduce Scal3R. This approach reformulates online reconstruction as multi-reference relative pose querying. We use lightweight learnable tokens, which make up about ~1% of the parameters, and inject them into a completely frozen backbone via asymmetric attention. This setup queries poses relative to multiple past keyframes. An online pose-graph optimization system with loop closure suppresses long-range drift. Scal3R reaches convergence in 8 hours on a single GPU. It reduces the average ATE by over 60% on KITTI compared to the online baseline. It also achieves state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes. Project page: https://linjohnss.github.io/scal3r/