Locally served AI models improve root cause detection in IT systems
TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceCryptography and SecurityEmerging TechnologiesMachine Learning
Summary
Finding problems in big computer systems is important but challenging, especially when using cloud-based AI that can be slow, costly, and risky for private data. The authors tested AI models that run on local powerful computers with three ways of asking questions, including one that looks up past problem records to help the AI. They found this lookup method made the AI more accurate and reliable, avoiding errors that made it say problems existed all the time. Their work helps companies analyze system errors safely and efficiently without relying on cloud services.
What this means in practice
- •For it operations teams: Identify root causes of incidents in on-premise IT systems more accurately and reliably using retrieval-augmented AI models.
- •For enterprise cloud architects: Reduce data privacy risks and network delays by deploying local AI models for log analysis instead of cloud-hosted services.
Authors
Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary
Abstract
Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk, network latency, and per-query cost that scale poorly with production log volumes. We present TriCalRAG, a benchmark evaluating open-weight LLMs served locally via vLLM on a single high-memory workstation GPU (NVIDIA RTX PRO 6000, 96GB) against a classical LSTM-based log anomaly detector (DeepLog), across four real, publicly available log datasets (BGL, HDFS, Thunderbird, OpenStack). We evaluate two open-weight models (Qwen2.5-14B, Mistral-Small) under three prompting strategies: zero-shot, few-shot, and retrieval-augmented generation (RAG) over a labeled incident history, reporting accuracy, precision/recall, and F1 with bootstrap 95% confidence intervals across 3 random seeds, alongside throughput and VRAM footprint. Our results show that RAG not only improves mean F1 by 0.10-0.27 over zero-shot prompting but, more importantly, substantially stabilizes model calibration: zero-shot prompting drives both models toward near-degenerate behavior (predicting "anomaly" on up to 100% of incidents on some datasets), while RAG keeps predicted-positive rates close to the true class balance in the majority of configurations. Mistral-Small achieves higher macro-averaged F1 than Qwen2.5-14B (0.644 vs. 0.560) but exhibits calibration failures in more configurations (7 vs. 5 of 12), while running at roughly half the throughput - indicating the better model choice depends on whether a deployment prioritizes peak accuracy or predictable behavior across prompting conditions. Ablations show batching scales throughput 41 times on a single card and that 4-bit quantization reduces latency 20% with no measurable accuracy loss. We release our benchmark harness, dataset splits, and evaluation code to support reproducible on-premise AIOps research.