Troubleshooting system improves internet service fault fixing with limited code
Agentic RCA for Internet-Scale Services Using Constrained Creativity
Networking and Internet ArchitectureArtificial Intelligence
Summary
Fixing problems in huge internet services is tricky because issues can be new and complex. The authors created a system called E4 that helps operators find and solve these problems more accurately and cheaply. E4 uses a special kind of controlled creativity, letting an AI generate only simple, safe programs that are easy to understand. This makes the system’s solutions both reliable and clear for humans to act on.
What this means in practice
- •For cloud operators: Identify and resolve outages or failures in large cloud services with more accurate and interpretable diagnostic insights.
- •For telecommunications engineers: Improve fault diagnosis in widespread telecom networks using cost-efficient automated reasoning with explainable outputs.
Authors
Sayan Sinha, Vipul Harsh, B. Aditya Prakash, Vyas Sekar, Hui Zhang
Abstract
System administrators of Internet-scale services need to resolve failure incidents to maintain reliability of such services. Ideally, we want a troubleshooting system to be: (1) expressive to known and unknown incidents with high accuracy; (2) cost efficient at scale; (3) explainable to provide actionable insights operators can act on; and (4) entail low effort from the operators. Unfortunately, most existing systems, including emerging LLM-assisted agentic workflows and structured frameworks for authoring diverse RCA algorithms fall short of achieving all four requirements. We present E4, a novel agentic system for troubleshooting for Internet-scale services. E4 embodies the paradigm of constrained creativity that combines the best of LLM-assisted automation and exploration with the explainability and efficiency of a structured approach. Instead of allowing an LLM agent to write arbitrary code or generate arbitrary responses, we provide the agent a restricted DSL to generate its response via simple loop-free data flow programs. This DSL, equipped with high level operators for troubleshooting, makes E4's output accurate, verifiable and explainable. On a mix of synthetic and real-world workloads, E4 achieves up to 62% better accuracy compared to state-of-the-art solutions, while providing more explainable responses at up to 12x reduced cost.