Uncertainty evaluation improves safety checks for lidar object detection
Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection
Computer Vision and Pattern RecognitionMachine LearningRobotics
Summary
Some cars use lidar sensors to detect objects, but the system can sometimes make mistakes. The authors studied how to set safe limits for when the system should trust its own detections based on uncertainty. They found that a safety limit proven in one situation might not hold when conditions change, like in rain or far distances. The paper proposes a test to check if the safety limit can be used safely in different conditions before relying on it.
What this means in practice
- •For automotive safety engineers: Verify when safety performance limits for lidar-based object detection remain valid across varying driving conditions and sensor uncertainties.
- •For autonomous vehicle testers: Apply statistical tests to confirm if machine learning detection thresholds retain their safety guarantees under new environmental conditions like rain or long distances.
Authors
Milin Patel, Rolf Jung
Abstract
For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, the same data gave not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors, thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.