ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed ReCal3R, a method to improve 3D reconstruction from video by carefully adjusting how much new information updates a stored scene. Their approach estimates how reliable each part of the scene is and uses that to control the update speed, avoiding corruption from noisy data while still learning from good observations. Testing on long video sequences showed better accuracy in pose, depth, and 3D scene quality without extra computational cost. This method can enhance existing reconstruction systems without retraining.
3D reconstructionrecurrent neural networkslearning ratepose estimationdepth estimationscene statetraining-free calibrationupdate pressurealignmentstreaming processing
Authors
Xinze Li, Yiyuan Wang, Pengxu Chen, Wentao Fan, Weifeng Su, Weisi Lin, Wentao Cheng
Abstract
Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7$\times$ reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.