Drive2gauss creates 4d driving scenes from multimodal generated video
Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos
Computer Vision and Pattern Recognition
Summary
Creating virtual driving environments that can be viewed from different angles is challenging. The authors introduce Drive2Gauss, a method that quickly turns videos with color images, depth, and motion into a 4D model of a scene without extra tuning. This allows for realistic views of a driving scene from new camera positions and helps fix videos. Their tests on driving data show promising quality in both generated videos and 3D scene reconstruction.
What this means in practice
- •For autonomous vehicle developers: Generate controllable 4D scene models from driving videos to test and visualize future scenarios without per-scene tuning.
- •For game developers: Create dynamic driving environments that allow changing viewpoints and camera angles from generated multimodal video data.
Authors
Tong Zhao, Zhiyuan Han, Lin Chen, Bowen Xie, Yunxi Qiao, Bohan Li, Weiqing Xiao, Gen Li, Shu Han, Xuyang Dai, Cheng Bi, Hao Zhao, Chaojian Li
Abstract
Generating reusable driving environments requires both controllable future observations and an explicit representation that supports rendering from different viewpoints, yet existing methods still recover explicit scenes from generated RGB videos through optimization for each sequence. To overcome these limitations, we present Drive2Gauss, a feedforward framework that converts jointly generated RGB, metric depth, and dynamic flow into dynamic 4D Gaussian scenes. A Multimodal Video Generator predicts the three modalities from reference multiview images and future control signals. Furthermore, we introduce a Gaussian Query Decoder that uses depth to initialize sparse 3D queries and flow to associate the queries across time, and integrates appearance information through spatiotemporal feature aggregation and iterative query refinement. The reconstruction stage operates in a single feedforward pass without optimization for each scene. Extensive results on nuScenes demonstrate that our approach attains an FID of 6.38 and an FVD of 12.43 for future RGB generation, a PSNR of 29.91 dB when reconstructing its generated observations, and supports both direct rendering under lateral camera shifts and downstream video fixing. Code and more visualizations are available at https://zttt04. github.io/Drive2Gauss_page/.