Simple large language models diagnose microservice failures better than complex agents
Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
Artificial IntelligenceDistributed, Parallel, and Cluster ComputingSoftware Engineering
Summary
Figuring out what caused a software failure in complex systems called microservices is hard, so some computer scientists build step-by-step programs to guide AI models in finding the cause. This paper shows that just letting simpler AI models freely investigate the evidence outperforms these complicated, hand-designed processes. The authors tested their approach on many real-world cases and found it was more accurate and often faster. They also discovered that improving the AI model itself helps reliability, while letting it do multiple reasoning steps improves its skill level.
What this means in practice
- •For software reliability engineers: Improve failure diagnosis accuracy by using simple LLMs without fixed procedures to quickly find root causes in microservice systems.
- •For cloud operations teams: Reduce time and effort in investigating service outages by adopting flexible LLM-based tools that better identify the key evidence for outages.
Authors
Tingzhu Bi, Xun Zhang, Jinglin Li, Ping Wang, Meng Ma
Abstract
Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model's calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65-77% on three mid-tier models), ahead of the published agents (16-53%) and the non-LLM methods (6-39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.