Driving view synthesis improves with region-based generative and reconstruction methods
PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis
Computer Vision and Pattern Recognition
Summary
Creating realistic videos of driving scenes from new angles is hard because existing methods either miss some parts or make views look inconsistent over time. The authors combine two types of techniques: one that reconstructs scenes based on real data and another that brings creative generation abilities. Their system divides each view into parts that should be kept, fixed, or newly generated. This approach results in clearer, more accurate driving videos even when the viewpoint changes a lot. Tests on Waymo data show their method performs better than previous ones.
What this means in practice
- •For autonomous vehicle developers: Generate realistic driving scenario videos from various viewpoints to improve simulation and testing of vehicle systems.
- •For simulated environment makers: Create high-fidelity driving videos with geometric accuracy for use in training or validating simulation platforms.
Authors
Sipeng He, Jialei Chen, Zhen Fang, Dongchun Ren, Feng Zhao
Abstract
Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation. Reconstruction-based methods leverage neural rendering to synthesize geometrically consistent views, but often exhibit diverse artifacts and missing content when the viewpoint deviates from the training trajectory. In contrast, generative models can synthesize realistic views along arbitrary trajectories from vehicle sensor data, yet often struggle to maintain temporal and geometric consistency across frames. To combine the strengths of both, we propose PRG-Fusion, a framework for driving view synthesis that uses reconstruction evidence to orchestrate generative priors across regions. Specifically, we extract region-wise degradation evidence from reconstructed driving scenes and convert it into Preserve, Repair, and Generate (PRG) labels. At inference, these labels serve as a unified routing policy for region-aware spatiotemporal synthesis, orchestrating 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively. We then follow a two-stage training paradigm, first establish geometric control from sparse LiDAR projections and subsequently learning appearance control from dense 3DGS renderings. Extensive experiments on Waymo demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis, with superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts.