Analyzing cyber-physical system failures with distance-weighted causality
Causal Analysis of Failures of Cyber-Physical Systems with Continuous Inputs
Robotics
Summary
Modern autonomous systems like self-driving cars are very complex, and it’s hard to fully check how they’ll behave before using them. When something goes wrong, it’s important to understand why. The authors improved a method that looks at causes of failures by considering how much input values need to change to prevent the failure, not just whether they change or not. This helps provide clearer explanations of failures in systems that deal with continuous data, like many autonomous machines. They tested their method on simulations and showed it works better than previous simpler approaches.
What this means in practice
- •For autonomous vehicle engineers: Explain failures in self-driving cars with a nuanced measure capturing continuous sensor inputs impact.
- •For control systems developers: Improve debugging of complex cyber-physical systems by quantifying how much input changes affect failures.
Authors
Tom Yaacov, David A. Kelly, Khen Elimelech, Hana Chockler
Abstract
Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, operate in increasingly complex environments while expected to maintain high reliability. The black-box nature of many of these systems makes it impossible to fully verify their behavior prior to deployment. When a failure occurs, it is important to analyze it to guide improvements to system robustness. A recent work introduced the novel concept of leveraging the framework of actual causality for CPS failure explanation. However, that work considered only a discrete, binary causal model that cannot capture continuous changes in system inputs. In this paper, we address these limitations by extending the actual causality framework with the notion of distance to account for the continuous characteristics of the system. Specifically, we introduce the novel concepts of distance-weighted actual cause and distance-weighted degree of responsibility, enabling causal influence to account for the magnitude of changes required to avoid a failure, rather than only its cardinality. While motivated by the continuous nature of CPSs, this constitutes a general extension of the theory of actual causality and enables more fine-grained analysis of continuous inputs in any causal model. Building on these notions, we present an efficient, practical, system-agnostic explanation-derivation algorithm that uses distance-weighted responsibility to guide the explanation search. We experimentally evaluate our approach across a number of environments from two autonomous-driving simulators with black-box neural controllers. The results show that the proposed extension provides more effective guidance than the current discrete approach.