PointForward V2 improves 3D driving scene reconstruction with query interaction
PointForward V2: Point-Aligned Interaction for Feedforward Driving Reconstruction
Computer Vision and Pattern Recognition
Summary
Reconstructing a full 3D view of driving scenes from multiple camera images over time is difficult because the system needs to combine lots of information efficiently. The authors improved a previous method called PointForward by letting 3D points interact through special compact data structures, helping the system share and update information across space and time without too much computing cost. They also use a way to tell apart points based on both position and time, so the system understands where and when each point belongs. Their method works better than previous ones on several big driving datasets.
What this means in practice
- •For autonomous vehicle engineers: Create more accurate 3D maps of driving environments by integrating multiple camera views over time with efficient point interactions.$Commercial implications: Improves autonomous vehicle perception software, enabling better environment understanding and safer navigation in real-world scenarios.
- •For robotics perception teams: Enhance real-time scene understanding in robots by using structured registers to manage spatio-temporal 3D data efficiently.
Authors
Xianqi Wang, Cheng Chi, Yuechuan Pu, Hongcheng Luo, Junda Cheng, Guang Chen, Hangjun Ye, Bing Wang, Xin Yang
Abstract
Feedforward reconstruction of dynamic driving scenes requires integrating observations across multiple views and time steps into a coherent 3D representation. PointForward established state-of-the-art feedforward driving reconstruction by lifting image observations into sparse 3D queries, demonstrating the effectiveness of point-aligned scene representations. However, these queries are largely processed independently, limiting information exchange across space and time, while direct global attention among tens of thousands of point queries is computationally prohibitive. In this work, we present \textbf{PointForward V2}, which advances point-aligned representation to point-aligned interaction. Our key idea is to mediate query interaction through compact, scene-structured registers. For the static background, frame-wise Static Scene Registers compress point information, communicate across frames, and propagate the aggregated context back to the queries. For each dynamic instance, queries from different frames are aligned in an object-centric canonical space and interact through instance-specific Dynamic Instance Registers. We further employ 4D positional encoding that combines aligned spatial coordinates with original timestamps, enabling spatially aligned yet temporally distinguishable interactions. In this way, PointForward V2 enables scalable spatio-temporal reasoning directly among lifted 3D queries without dense global attention. Experiments on Waymo, nuScenes, and Argoverse 2 demonstrate state-of-the-art performance across large-scale driving benchmarks.