AI summaryⓘ
The authors created a large dataset called Realsee3D with both real and synthetic indoor panoramic images that include depth information and accurate measurements. They then developed Argus, a neural network that uses this dataset to quickly and accurately reconstruct 3D scenes from these panoramic views. Argus solves the problem of picking a wrong reference point, which can mess up the 3D map, by learning to choose the best view to anchor the scene. They also improved the learning by breaking down the 3D reconstruction into smaller, understandable steps with clear guidance. Their method outperforms previous approaches in estimating camera position, depth, and reconstructing 3D point clouds on their benchmark data.
panoramic RGB-D data3D reconstructionmetric annotationsfeed-forward networkcamera pose estimationdepth estimationpoint cloud reconstructioncovisibilitycoordinate anchoringmulti-task learning
Authors
Xi Li, Linyuan Li, Yan Wu, Tong Rao, Kai Zhang, Xinchen Hui, Cihui Pan
Abstract
Metric feed-forward 3D reconstruction for panoramic data remains under-explored due to the lack of large-scale panoramic RGB-D training data. We present Realsee3D, a hybrid dataset of 10K indoor scenes (1K real, 9K synthetic) with 299K panoramic viewpoints and precise metric annotations, and Argus, a feed-forward network trained on it for metric panoramic 3D reconstruction. In the sparse unordered capture setting of Realsee3D, a poorly chosen coordinate anchor can cause global pose drift. Argus addresses this with a learned covisibility module that selects the geometrically optimal reference view to anchor the metric world frame. To further improve multi-task learning, we decompose the bidirectional pixel-to-world mapping into interpretable sub-steps with per-step supervision and cross-coordinate joint constraints, reinforcing geometric consistency across prediction branches. On the Realsee3D benchmark, Argus achieves state-of-the-art metric performance in camera pose estimation, depth estimation, and point cloud reconstruction. Project page: https://argus-paper.realsee.ai.