Surface aware encoding improves multi view image synthesis quality
HRoPE: Homography Relative Positional Encoding
Computer Vision and Pattern Recognition
Summary
Creating new views of a scene from multiple images is hard because matching details across different viewpoints can be tricky. The authors suggest a new way to help computers understand the 3D shapes in a scene by imagining each small patch as a tilted piece of a 3D plane. This helps the system line up and compare images more accurately, leading to better results when generating new views. Their method doesn’t need exact 3D measurements beforehand and performs better than previous methods on real-world image sets.
What this means in practice
- •For augmented reality developers: Generate realistic new views of real-world scenes by better aligning image features from multiple cameras without requiring exact 3D scans.$Commercial implications: Enables AR apps to create seamless perspective changes and immersive experiences by improving multi-view image synthesis with learned geometry.
- •For visual effects artists: Create novel camera angles from multi-view footage by accurately warping and blending image patches based on inferred surface geometry.
Authors
Huachen Gao, Yang Yang, Xiaoqian Liang, Xingyu Chen, Youjia Zhang, Qi Zhang, Baorui Ma, Anpei Chen, Wei Yang
Abstract
Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry. Such sensor-centric formulations, however, describe only where a token is observed from, leaving the rich surface geometry of the scene region depicted by the token. In this paper, we advocate a shift toward surface-aware positional encoding that explicitly grounds patch tokens in the geometry of physical surfaces. We introduce HRoPE, which represents the local geometry of each patch as a slanted 3D plane, following the slanted support-window assumption of classical multi-view stereo. We inject token-level, plane-induced homographies into attention as relative transformations. This equips the transformer with a geometric prior for rectifying perspective distortions and aligning patch features across viewpoints. Our encoding requires no ground-truth geometry: the plane parameters can be predicted end-to-end under a view-synthesis objective alone, or derived from foundation models. Extensive experiments on RealEstate10K, CO3Dv2, and DL3DV show that our best variant of HRoPE consistently outperforms the evaluated geometric positional encodings for novel view synthesis. Extensive ablations and analysis show the effectiveness of the proposed encoding. Please refer to our project page at: https://gaohchen.github.io/HRoPE