Technique improves camera localization using neural fields and ray prediction
NaCR: Visual Localization via NeRF-aided Camera Ray Regression
Computer Vision and Pattern Recognition
Summary
Visual localization helps devices understand where they are by looking at pictures. The authors focus on a new approach that predicts where a camera is pointing from small parts of an image but found it was not very accurate. They realized that a neural rendering method called NeRF, which creates pictures from camera views, could help improve this by providing extra training data and feedback. Their method combines both ideas to better predict camera directions, making location estimates more precise in a variety of environments.
What this means in practice
- •For virtual reality developers: Create more accurate user positioning in VR systems by improving camera localization techniques using neural rendering and ray prediction.
- •For robotics navigation teams: Enhance robot localization accuracy indoors and outdoors by integrating neural field guided camera ray regression methods.
Authors
Yesheng Zhang, Xiang Dai, Xu Zhao, Chongyang Zhang
Abstract
Visual localization (VL) is a fundamental technology for vision applications such as virtual reality. Recently, a novel VL paradigm, Camera Ray Regression (CRR), has emerged, which maps 2D image patches to 3D camera rays, but its accuracy is limited. To improve CRR accuracy, we notice a compelling duality: the inverse of this mapping is inherently performed by the novel view synthesis model, \ie, Neural Radiance Fields (NeRF). While NeRF renders image patches from camera rays via differentiable ray marching, CRR predicts the rays from image patches. Motivated by this complementary relationship, we propose NeRF-aided Camera Ray Regression (NaCR), a unified framework that seamlessly bridges NeRF and CRR at the ray level. First, NaCR incorporates three simple yet effective enhancements into the CRR baseline. Second, leveraging a pre-trained NeRF, NaCR augments the training data by synthesizing novel views tailored for efficient, patch-level consumption. Finally, exploiting the differentiability of NeRF, NaCR forms a closed-loop supervision pipeline where photometric rendering errors are back-propagated to optimize the predicted camera rays. To ensure stable convergence within the highly non-convex image space, we introduce a two-stage training curriculum. Extensive experiments across indoor and outdoor benchmarks demonstrate that NaCR achieves competitive accuracy. Comprehensive ablation studies validate the efficacy of each proposed component.