Ray-SCR improves scene localization across diverse camera types
Ray-SCR: Generalizing Scene Coordinate Regression Across Camera Models
Computer Vision and Pattern Recognition
Summary
Localizing where a photo was taken is hard when different cameras show the same scene in very different ways. The authors present Ray-SCR, a method that makes it easier to guess scene coordinates accurately no matter the camera type, whether it’s a normal lens, fisheye, or omnidirectional camera. They do this by using the direction of rays from the camera, which helps the system understand how different lenses see the scene. This approach works well on different datasets and camera types, making it faster and more reliable to locate images in various settings.
What this means in practice
- •For robotics engineers: Enable robots to precisely localize themselves using cameras with widely different lenses in the same environment without retraining on each camera type.
- •For augmented reality developers: Use a single 3D scene map for accurate localization of AR content regardless of the user’s device camera model, simplifying app maintenance.
Authors
Jaemin Cho, Oleksii Nasypanyi, Francois Rameau
Abstract
Among visual localization strategies, scene coordinate regression (SCR) has emerged as an attractive approach for its compactness, accuracy, and speed. Despite these advantages, SCR models generalize poorly across substantially different camera models, such as omnidirectional or strongly rectified imagery. This limitation arises because, changes in field of view, lens distortion, or rectification-induced resampling can make the same scene content appear differently depending on its pixel position, degrading localization accuracy. To cope with this limitation, we introduce Ray-SCR, a lightweight extension to existing SCR architectures that conditions local visual features on their calibrated viewing rays and refines neighboring ray-conditioned features with a shallow local Transformer. We further formulate supervision directly in ray space and use sparse SfM points as metric anchors that link observations across views and camera models. Taken together, these components enable a single scene representation built from a panoramic mapping sequence to accurately localize query images captured with very different cameras, including pinhole, fisheye, and omnidirectional cameras. We evaluate Ray-SCR across multiple SCR architectures and on datasets with pinhole, fisheye, and omnidirectional cameras. The proposed ray conditioning substantially improves generalization across cameras and leads to state-of-the-art localization performance in these challenging cross-camera settings. The project page is available at https://jaeminch0.github.io/ray-scr-generalizing-scene-coordinate-regression-across-camera-models/.