G-ray improves multi-view vision transformer geometry across cameras
G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity
Computer Vision and Pattern Recognition
Summary
Transformers used to combine images from multiple cameras struggle when camera settings differ, like having different fields of view. The authors introduce G-ray, a method that encodes camera rays using angles instead of image positions, making the way the Transformer understands positions consistent across diverse cameras. This helps 3D reconstruction models work better even when cameras vary and lets them handle mixed camera types without extra training. The method also improves generating new views of scenes under changing camera angles and zoom levels.
What this means in practice
- •For computer vision engineers: Improve 3D reconstruction models to handle images from multiple cameras with different fields of view or projection types without retraining.
- •For augmented reality developers: Enhance rendering of new viewpoints in AR applications when cameras with varying angles and zooms are used together.
Authors
Shuo Zhang, Xin Su, Wei Wang, Jun Liu, Xinrui Zeng, Yongsen Chen, Chenjie Wang, Guibo Zhu, Jinqiao Wang, Bin Luo, Liangpei Zhang
Abstract
We study relative position encoding for multi-view vision Transformers under camera heterogeneity, including varying fields of view (FoVs) or projection models. Existing rotary relative position encodings commonly use image-plane positional coordinates, producing projection-dependent relative phases and inconsistent geometric cues for cross-projection attention. We introduce G-ray, a ray-level relative position encoding whose rotary phases are parameterized by camera-local ray angles. The same camera-local ray pair induces the same relative phase across projections, providing projection-invariant positional consistency. G-ray can be used directly or integrated with existing encodings, retaining complementary geometric cues without additional learned parameters. We validate G-ray in three host encodings, RoPE, GTA, and RayRoPE, across 3D reconstruction and novel-view synthesis (NVS). Across three heterogeneous 3D reconstruction benchmarks at 50 views, G-ray leads all six averaged metrics and reduces mean pointmap relative error by 45.8% over MapAnything, with calibration supplied to both. Trained exclusively on homogeneous pinhole images, the 3D reconstruction model handles mixed pinhole and non-pinhole inputs without retraining and remains competitive on homogeneous pinhole 3D reconstruction protocols. For NVS, GTA and RayRoPE improve with G-ray under joint viewpoint and FoV variation. The project's webpage is available at https://g-ray-project.github.io/.