Autoware universe shows safety gaps in autonomous driving tests
Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles, Part II: Assessment Results
Robotics
Summary
Autonomous cars need to be very safe before lots of people use them. The authors tested a popular open-source self-driving system called Autoware.Universe on both simulated roads and a physical test track. They found that this system met less than half the important safety behaviors and crashed much more often than average human drivers. The tests also revealed new unsafe situations the system couldn't handle well. This shows it’s important to carefully check how self-driving cars behave before letting them drive widely on real streets.
What this means in practice
- •For automotive safety engineers: Identify specific behavioral safety gaps in level 4 autonomous systems for targeted improvements before release.
- •For autonomous vehicle testing teams: Use the validated third-party testing framework to assess advanced driving systems under realistic and simulated conditions.
Authors
Henry X. Liu, Tinghan Wang, Xintao Yan, Haowei Sun, Zhijie Qiao, Kenneth Boyd, Shuo Feng, Greg Stevens, Greg McGuire
Abstract
Third-party evaluations of autonomous vehicle (AV) safety can play a vital role in improving public acceptance, building consumer confidence, and establishing effective safety standards. In Part I of this study, we propose a dedicated third-party testing initiative for systematically evaluating AV behavioral safety. In this paper, we validate our proposed framework using Autoware.Universe, an open-source Level 4 Automated Driving System (ADS), tested both in simulated environments and on the physical test track at the University of Michigan's Mcity Testing Facility. The results indicate that Autoware.Universe possesses 6 out of 14 behavioral competencies and exhibited a crash rate of 3.01x10^-3 crashes per mile, approximately 1,000 times higher than the average human driver crash rate. During the tests, we also uncovered a number of unknown unsafe scenarios for Autoware.Universe. These findings underscore the necessity of behavioral safety evaluations for improving AV safety performance prior to widespread public deployment.