AI summaryⓘ
The authors studied how well self-driving car systems handle different kinds of disruptions using three testing methods: offline analysis, hardware simulation, and real driving tests. They tested 72 types of changes to camera and LiDAR sensors on two types of driving systems. Their findings show that simple tests on recorded data don't always predict how a car will behave in real driving, especially for camera-based systems where small issues can cause big problems on the road. They also found that real-time testing is important and that checking only the model’s accuracy isn’t enough to know if the system is truly robust. This suggests real-world testing is crucial for understanding how self-driving cars perform under tough conditions.
Autonomous Driving SystemsPerturbation TestingCamera SensorsLiDAROffline Model AnalysisHardware-in-the-LoopClosed-Loop TestingEnd-to-End Vision ModelsModular Perception and PlanningSystem-Level Evaluation
Abstract
Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end vision-based driving model and a modular LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.