Small language models improve root cause analysis with historical frequency data
Fault-Free Profiles and Decision-Level Training for Small-Model Root Cause Analysis
Software Engineering
Summary
Finding the true cause of problems in complex software systems is tricky because many signals can look unusual but are actually normal. The authors found that small language models often pick wrong causes because they miss context about what is normally unusual. They created a system that teaches these models to understand how often unusual signals happen when everything is fine, helping them avoid false alarms. This makes the models better at correctly finding real problems using less computing power and protecting privacy.
What this means in practice
- •For site reliability engineers: Improve identification of true failure causes in service monitoring systems using small language models that consider historical normal signal behavior.
- •For cloud operations teams: Reduce false-positive alerts in cloud infrastructure by integrating historical anomaly frequency profiles into root cause analysis tools based on small models.
Authors
Kyoohyun Lee, Woohyeok Park, Taeyoon Kim, Sungyong Baik, Kyungyong Lee
Abstract
Root cause analysis (RCA) in microservice systems requires distinguishing failure sources from routine fluctuations and propagated symptoms. Small language models (SLMs) offer a practical solution under cost and privacy constraints, and recent approaches improve their capabilities through structured diagnostic procedures. However, performing diagnostic analysis does not ensure correct root-cause identification. Our empirical study shows that final decisions often favor large anomaly scores despite the model's own propagation analysis suggesting otherwise. Moreover, the input does not reveal whether these apparently anomalous signals are common during normal operation. These findings motivate our system, which improves how SLMs assess evidence and select root causes through Historical Frequency Profiling and Biased-Decision Correction. Historical Frequency Profiling quantifies how often metric and log signals are flagged as anomalous in the fault-free period, providing context for distinguishing routine deviations from potential failure evidence. Biased-Decision Correction identifies points where the model reaches different conclusions from the same initial analysis and uses preference optimization to favor more accurate diagnoses. Against baselines including the 14B OpsAgent and RCA-Agent with commercial models, our system with a 4B model achieves the highest or tied-highest accuracy on all four datasets, reaching 48.7% on Bank compared with 18.5% for OpsAgent.