Deep causal models improve root cause analysis with hidden factors
Beyond Conditional Independence: Root Cause Analysis with Deep Causal Models
Machine LearningArtificial Intelligence
Summary
When something goes wrong in a complex system, it's important to find the exact cause quickly. Existing methods often assume no hidden influences, which limits their accuracy. The authors developed a new method using deep causal models that can handle hidden or unknown factors affecting the system. Their approach better identifies causes of problems by modeling how different parts of the system influence each other and changes when faults occur. Tests show it works better than other leading methods on simulated and real-world systems.
What this means in practice
- •For cloud infrastructure teams: Identify failing services in microservice architectures even when hidden system interactions exist.
- •For industrial automation engineers: Pinpoint root causes of faults in complex machinery where not all factors can be directly measured.
Authors
Md Musfiqur Rahman, Kenneth Lee, Ziwei Jiang, Padmaja Jonnalagedda, Ruocheng Guo, Murat Kocaoglu
Abstract
Root cause analysis (RCA) is a critical problem in many real-world scenarios. RCA enables the identification of faulty or failing mechanisms in a system by comparing anomalous observations with corresponding reference (i.e., regular) observations. However, existing approaches rely either on heuristic methods or on conditional independence tests with a strong unconfoundedness assumption, and thus fail to exploit other complicated distributional constraints in the presence of latent variables. To relax these assumptions, we model the underlying system as a causal model and the anomalous system as a change in the structural functions of the same causal model. Specifically, to handle unobserved confounders, we establish an implicit connection between distributional constraint testing and root cause analysis. To adapt our approach to data generated from arbitrary causal models, we employ the deep causal model (DCM) framework, in which we design the causal model using neural networks. Finally, we illustrate how our method, RCA-DCM, can utilize different levels of partial graphical knowledge to perform RCA. We evaluate RCA-DCM against state-of-the-art baselines on simulated datasets, a physics-based causal chamber and two micro-service applications. RCA-DCM improves top-1 accuracy over the strongest baseline on both Sock Shop (0.880 vs. 0.752) and Online Boutique (0.776 vs. 0.712), and when the true root cause in the causal chamber is unobserved and acts as a latent confounder, it recovers the exact root-cause set more often than any competing method (perfect recovery rate (PRR) 0.846 vs. 0.731).