Scenario orchestrator improves testing of autonomous driving policies
Scenario Orchestration for Closed-loop Policy Evaluation of Autonomous Driving
Robotics
Summary
Testing self-driving car programs is tricky because the behavior of other cars on the road needs to respond realistically to the self-driving car's actions. The authors introduce a system that actively adjusts how other cars drive during simulation tests, making sure the test scenarios unfold as intended even when the self-driving car behaves unexpectedly. This method helps separate cases where the test scenario didn’t happen as planned from when the self-driving car actually failed. Their experiments show this approach leads to more reliable and repeatable tests compared to traditional fixed or reactive methods.
What this means in practice
- •For autonomous vehicle development teams: Test self-driving car algorithms in simulations that adapt other vehicles to ensure intended interaction scenarios occur despite unexpected behaviors.
- •For simulation platform builders: Incorporate adaptive scenario orchestration to improve scenario fidelity and reproducibility in traffic simulations supporting autonomous vehicle testing.
Authors
Frieda Rong, Zhaoyue Wang, Veer Kapadia, Vishaka Rajani, Aren A. Babikian, Xujie Si, Mahsa Golchoubian, Raquel Urtasun, Igor Gilitschenski
Abstract
Scenario-based simulation is central to evaluating autonomous-driving policies, but a scenario scripted before execution (i.e., with predetermined, fixed actor routes and speeds) may not preserve its intended interaction once the tested ego policy responds in closed loop. Scripted scenarios cannot adapt to unanticipated ego behavior, while learned traffic actors may remain reactive without reliably realizing a particular scenario intent. We introduce a scenario orchestrator that continuously monitors the joint traffic state and adapts non-ego actors through re-timing, re-routing, and role reassignment for creating a specific scenario intent. At each step, the orchestrator solves a constrained optimization problem that preserves the intended interaction while limiting interventions and maintaining traffic feasibility. We further distinguish scenario realization from policy failure, allowing a test to first be judged based on whether the intended interaction occurred and then by how the ego policy responded. Experiments in the CARLA simulator across multiple scenario families and driving policies show that orchestration realizes targeted interactions more reliably than scripted and learned actor-level baselines, particularly under variations in initial conditions and ego behavior. These results support closed-loop scenario orchestration as a tool for controlled and reproducible policy evaluation.