WorldRoPE improves multi-view attention with 3D uncertainty modeling
WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Usually, computers look at pictures separately and have trouble matching the same point seen from different angles. This paper introduces WorldRoPE, which thinks about the 3D location of parts of images with uncertainty. By representing image patches as blurry 3D shapes instead of points, it helps the computer better understand where they might appear in other views. This makes the attention process in Transformers more reliable when combining multiple views, improving 3D reconstructions and depth predictions.
What this means in practice
- •For computer vision teams: Improve 3D reconstruction and novel view synthesis by integrating uncertainty-aware multi-view attention into Transformer models.
- •For augmented reality developers: Enhance spatial understanding in AR by reliably combining multiple camera views using probabilistic 3D positional encodings.
Authors
Mijeong Kim, Kyujin Lim, Bohyung Han
Abstract
Rotary Positional Embedding (RoPE) is widely used to encode relative positions in Transformers. In multi-view settings, however, patch coordinates are defined independently in each image: the same 3D point can therefore have different coordinates across views, making them insufficient for establishing cross-view correspondences. We introduce WorldRoPE, a probabilistic world-centric RoPE for multi-view attention. At each layer, WorldRoPE back-projects each patch token into a set of volumetric Gaussian primitives in a shared world frame. These primitives represent multiple plausible 3D supports for a token, jointly capturing alternative depth hypotheses, depth uncertainty, and the finite spatial extent of its patch frustum. When a key token's support is projected into a query view, it induces a distribution over where that key may appear in the query image. WorldRoPE integrates the rotary operator over this distribution in closed form, combining rotations at the projected primitive centers according to their mixture weights with covariance-dependent attenuation. This makes attention less sensitive to uncertain key locations in the query view, without rasterizing or warping token features. The formulation reduces to deterministic RoPE when the support collapses to a point. Across layers, the predicted geometry guides cross-view attention, while the updated features refine the geometry used at the next layer. Experiments on novel view synthesis, depth estimation, and 3D geometry reconstruction demonstrate the effectiveness of WorldRoPE for multi-view Transformers.