ReconDreamer# improves 3D driving scene reconstruction with self-evolving synergy
ReconDreamer#: Self-Evolving Closed-Loop Synergy of Reconstruction and Generation for Dynamic Driving Scenes
Computer Vision and Pattern Recognition
Summary
Reconstructing dynamic driving scenes in 3D from sensor data can be inaccurate because generated scenes do not perfectly match real-world observations. The authors introduce ReconDreamer#, a new method where the generation and reconstruction parts of the model work together and improve each other over time, closing the gap between synthesized and real data. They also add a model to improve ground surface details, which helps render higher-quality scenes. Tests on multiple driving datasets show that this method produces more accurate and realistic 3D reconstructions than previous versions.
What this means in practice
- •For autonomous vehicle developers: Improve 3D reconstruction accuracy of dynamic scenes to enhance perception and prediction modules in self-driving cars.
- •For simulation software engineers: Generate more realistic and higher fidelity dynamic driving scenarios for testing and training autonomous driving systems.
Authors
Songling Liu, Guosheng Zhao, Xiaofeng Wang, Chaojun Ni, Guan Huang, Jiwen Lu, Xingang Wang, Zheng Zhu
Abstract
Dynamic driving scene reconstruction has made remarkable strides in recent years by leveraging generative priors, particularly in the quality of novel trajectory rendering. Nevertheless, a significant gap remains between the generated data and real-world sensor observations, severely hindering further improvements in reconstruction fidelity. In this paper, we present ReconDreamer#, a self-evolving framework that synergizes reconstruction and generation for dynamic driving scenes, which substantially bridges the gap between synthesized novel trajectory data and recorded real-world observations. Within this self-evolving closed-loop framework, the generative and reconstruction models mutually reinforce each other through iterative refinement, ultimately yielding a significant enhancement in the 3D representation capability of the reconstruction model. Additionally, we introduce a geometry-prior-based adaptive ground model to further enhance the rendering quality of the reconstruction model. To validate the effectiveness of the proposed method, we conduct comprehensive evaluations on Waymo, nuScenes, PandaSet, and EUVS benchmarks. Specifically on Waymo, at a 6-m lateral shift, ReconDreamer# improves NTA-IoU and NTL-IoU by 8.8% and 3.0%, respectively, and reduces FID by 15.1% relative to ReconDreamer++.