CoDrive generates consistent multi-vehicle driving videos with precise trajectory control

CoDrive: Cross-Vehicle World-Consistent Video Generation with Precise Trajectory Control for Cooperative Driving

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Driving involves many vehicles interacting at the same time, but most computer models only simulate from the viewpoint of one car. The authors created CoDrive, which can generate videos showing multiple vehicles in the same driving scene, making sure they see a shared, consistent world. CoDrive uses special attention mechanisms to understand relationships within each vehicle’s views and between different vehicles. It also controls their camera paths precisely, so each vehicle’s video matches what would happen in the real world. The authors tested CoDrive on various scenarios and found it improved control and consistency without losing image quality.

What this means in practice

Authors

Yu Meng, Baining Zhao, Junta Wu, Tengfei Wang, Rongze Tang, Haiyu Zhang, Wenqiang Sun, Chen Gao, Zhibo Chen, Xinlei Chen, Yong Li, Xiao-Ping Zhang, Chunchao Guo

Abstract

Real-world driving is inherently multi-agent, yet most existing driving world models generate observations from a single ego vehicle. Independently extending them to multiple vehicles does not ensure that different agents observe a consistent shared world. We present CoDrive, a cross-vehicle, multi-view driving video generation framework that jointly generates observations of vehicles sharing the same dynamic scene with precise camera-trajectory control. CoDrive interleaves local self-attention, which models spatiotemporal dependencies among the views of each vehicle, with global self-attention, which enables information exchange and consistency modeling across vehicles. To explicitly encode their spatial relationships, all camera trajectories are represented in a shared world coordinate system and injected into the attention layers through projective relative positional encoding. We further adopt a progressive mixed-task training strategy that combines large-scale real-world single-agent data with synthetic cross-agent interaction data, allowing the model to benefit from real-world appearance distributions while learning cross-agent consistency from simulation. For systematic evaluation, we introduce CoDrive-Bench, a benchmark covering real and synthetic multi-vehicle scenarios and evaluating trajectory controllability, scene geometry consistency, and instance-level consistency. Experiments show that CoDrive improves trajectory controllability and cross-agent geometric and instance consistency while maintaining competitive visual quality.
View PDFOpen arXiv