ExceptionDrive benchmark measures autonomous driving safety in rare hazards
ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving
RoboticsArtificial Intelligence
Summary
Autonomous cars can handle everyday driving but may struggle with rare, dangerous situations. The authors created ExceptionDrive, a way to add tricky hazards into real driving scenes to see how well self-driving car planners react. They also made new ways to measure if the planners avoid dangers and keep safe distances. Additionally, they built a Reminder Agent that helps spot hazards and suggest safety actions without needing detailed instructions.
What this means in practice
- •For autonomous vehicle developers: Assess and improve how planning algorithms handle rare and safety-critical driving hazards using ExceptionDrive’s challenging, edited scenes and new safety metrics.
- •For automotive safety engineers: Use the Reminder Agent to automatically detect hazards from vehicle sensors and suggest high-level safety strategies without custom labeling.
Authors
Ziyi Luo, Zhe Sun, Yehao Lu, Lei Zhou, Lisheng Wu, Xuewei Li, Zequn Qin, Xi Li
Abstract
Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDrive, a counterfactual planning benchmark that uses VLM-assisted screening, localized multi-view editing, and quality auditing to insert hazards into real nuScenes scenes while preserving their context. Its 21 tasks span six safety families and define hazard or conflict regions, local safety constraints, and acceptable responses. Because hazard insertion can invalidate the recorded human trajectory, our reference-free protocol evaluates edited predictions using Unsafe Rate (UR), Hazard Clearance Compliance (HCC), Hazard Proximity Response (HPR), and Counterfactual Trajectory Shift (CTS), which measure core-region intrusion, clearance compliance, clearance relative to a prescribed margin, and counterfactual trajectory change. Seven representative planners frequently intrude into hazard regions or provide insufficient clearance. We also develop a Reminder Agent that, without sample-specific task labels, converts visual evidence and the shared taxonomy into structured records of hazard presence, type, and a recommended high-level strategy. The agent neither predicts trajectories nor controls the vehicle; its records guide a VLM-based decision agent. In zero-shot experiments, the reminders improve strategy accuracy and reduce under-warning.