Hierarchical pose graphs improve 3D reconstruction from long videos
HiP3R: Hierarchical Pose Graphs for Long-Horizon 3D Reconstruction
Computer Vision and Pattern Recognition
Summary
It is hard to accurately stitch together thousands of frames from a long video into a single 3D scene because of inconsistent local maps and duplicated camera positions. The authors created HiP3R, which solves this by first arranging video chunks to fix overall scale and layout, then refining each camera frame within this global map. This approach balances efficiency and detail, improving camera paths and 3D models without needing extra training. It works for different video processing methods and has been tested on multiple benchmarks.
What this means in practice
- •For robotics developers: Generate globally consistent 3D maps from long robot navigation videos by combining chunk-level and frame-level pose optimization.
- •For augmented reality engineers: Improve stability and scale accuracy of 3D scene reconstruction from continuous camera streams in AR devices without retraining models.
Authors
Zunnan Xu, Shangchen Zhou, Yihang Luo, Chen Change Loy
Abstract
Feed-forward models accurately estimate camera motion and 3D geometry from short video windows. However, computational and memory costs prevent direct inference on sequences containing thousands of frames. Long videos must therefore be processed as overlapping chunks. This division produces independent local reconstructions with inconsistent coordinate frames and duplicate estimates of shared cameras. Achieving global consistency requires propagating constraints from adjacent overlaps and distant revisits across the sequence. This requirement naturally motivates pose graph optimization. However, the choice of graph level creates a trade-off. Chunk-level graphs efficiently resolve scale drift and close large loops. Yet, their rigid transformations cannot correct internal errors or conflicting frame predictions. Conversely, frame-level optimization offers local flexibility, but scaling it globally is computationally expensive and can break metric scale. HiP3R resolves this trade-off using a hierarchical pose graph. HiP3R first optimizes a robust chunk-level $\mathrm{Sim}(3)$ graph to establish the global layout and scale. It then holds this layout fixed and estimates one shared $\mathrm{SE}(3)$ camera per frame, turning overlapping window predictions into a single trajectory. Experiments across multiple benchmarks demonstrate the effectiveness of HiP3R in improving camera trajectories and 3D reconstruction. Our method applies to both chunk-based and streaming backends without additional training or fine-tuning. The codes are available at https://kkakkkka.github.io/HiP3R.