DreamStream improves simulation for evaluating driving policies more accurately
DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving
RoboticsComputer Vision and Pattern Recognition
Summary
Testing how self-driving car software behaves is hard because simulators don’t perfectly match the real world, especially in how the scene looks and changes. The authors created DreamStream, which generates video simulations that keep the important details self-driving systems need to make decisions. They also made a new way to measure how close these simulations are to reality based on what policies actually observe. This helps find real weaknesses in self-driving software that older tests missed.
What this means in practice
- •For autonomous vehicle developers: Use DreamStream to test and improve driving policies by exposing subtle failure modes under realistic interactive scenarios with environmental variations.
- •For computer vision engineers: Evaluate and refine perception systems with generated video streams that preserve critical scene features relevant for decision making.
Authors
Ziyang Leng, Sicheng Mo, Seth Z. Zhao, Haoyuan Cai, Yu Zeng, Rowan McAllister, Bolei Zhou
Abstract
Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene features a policy relies on to make decisions. Existing platforms, however, exhibit a sim-to-real visual gap that corrupts policy perception, undermining their ability to assess a policy's closed-loop decision-making. To this end, we propose DreamStream, a generative, closed-loop simulator that achieves policy-oriented fidelity using a simulator-grounded autoregressive video model. Our video model is distilled from a large pretrained video model via traffic layout guidance, varying visual appearance while preserving policy-relevant features such as scenario layout and the temporal consistency of dynamic objects. We further observe that perceptual metrics like FID misrank how well these features are preserved. To tackle this, we introduce FD$π$, a new multi-representation metric that measures the sim-to-real gap as the Fréchet distance over scene-context features from public E2E policies. Under FD$π$, DreamStream improves over the strongest prior closed-loop simulator by $1.6\times$ on nuScenes and $4.7\times$ on NAVSIM, and induces the least perturbation to policy's perceptual observability. Based on DreamStream, we construct Navhard-CL benchmark, which turns non-reactive real-world benchmark NAVSIM into interactive testing environments with adversarial driving behaviors and weather variations. This benchmark exposes many failure modes of driving policies, such as scorer bias and lack of recovery behaviors, that prior closed-loop benchmarks overlook. Code and data are available at https://github.com/VAIL-UCLA/DreamStream.