Planner-specific training cuts collisions in self-driving models
Model-Aware Generation of Safety-Critical Scenes for Robust End-to-End Driving via Distilled Planner Surrogates
RoboticsComputer Vision and Pattern Recognition
Summary
Self-driving car models can often perform well in general but still crash in tricky situations. The authors found that many crashes come from the planning part of the system rather than sensing. They showed that because rare, dangerous situations are so uncommon in real driving data, planners don't learn well how to handle them. Their method creates tricky scenarios tailored to each planner by simulating challenges and safe responses, then uses these scenes to retrain the planner. This approach reduced crashes by about half on challenging test cases.
What this means in practice
- •For autonomous vehicle developers: Improve self-driving planners by generating and using rare safety-critical scenes specific to their planning algorithms, reducing collision rates in simulations.
- •For automotive simulation engineers: Automate creation of diverse challenging driving scenarios aimed at specific planners to better test and fine-tune self-driving models.
Authors
Khaled M. Seyam, Julian Wiederer, Samar Fathalah, Markus Braun, Bin Yang
Abstract
End-to-end driving models perform well on standard benchmarks but collide frequently in safety-critical scenarios. We present a failure-mode analysis on NeuroNCAP which shows that 95.9% of DiffusionDrive's collisions are planning failures despite reliable detection, pinpointing the bottleneck in planning rather than perception. We attribute this to a fundamental data imbalance, as safety-critical interactions are rare in real driving logs, so planners are hardly ever trained on them. We close this gap with a novel model-aware framework that generates safety-critical training scenes against a specific planner. Each planner is attacked at the bird's-eye-view (BEV) level through a differentiable surrogate distilled from its own planning head. This keeps the expensive video renderer out of the attack loop. We extract 50 scenes per planner, each pairing an attack with a safe escape trajectory that serves as the fine-tuning label. A fully automated quality-diversity search generates both the attacks and the escapes, so that their variety is not specified by hand. Only these scenes are rendered to fine-tune the planner, which reduces the computational cost significantly. Fine-tuning on these scenes roughly halves the closed-loop collision rate of two planners on NeuroNCAP (SparseDrive 28.0% to 12.2%, DiffusionDrive 37.5% to 19.2%) while largely preserving normal driving. A cross-planner transfer experiment shows that the gain comes from targeting the specific planner. Each planner improves most on the scenes generated against itself, and DiffusionDrive fine-tuned on SparseDrive's scenes retains only 6.7 of its 18.3-point collision-rate reduction.