Large language models vary in log anomaly detection effectiveness and cost
Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustness
Machine LearningCryptography and Security
Summary
Detecting unusual errors in computer logs is important but tricky. The authors studied how different ways of using large language AI models affect their ability to find these errors, how much computing power they need, and how well they work when the data is messy. They found that some training methods work better than others, bigger models don’t always help the same way for every dataset, and cheaper model versions still perform well. Their work helps people choose the best model setup for spotting log problems efficiently and reliably.
What this means in practice
- •For it operations teams: Choose optimal large language model settings to detect system log anomalies efficiently under varying conditions.
- •For cloud service providers: Optimize model deployment configurations to balance accuracy and computational cost when monitoring cloud infrastructure logs.
Authors
Bin Li, Dongdong Wang, Siyang Lu
Abstract
Large language models (LLMs) have demonstrated promising performance in log anomaly detection, yet how their adaptation strategies, architectures, and deployment configurations affect detection effectiveness remains insufficiently understood. To investigate these factors, we conduct a systematic empirical analysis across three public log datasets, examining different adaptation strategies, model architectures, parameter scales, and quantization settings. Our results reveal substantial performance differences across adaptation strategies, while model scaling yields varying detection gains across datasets. We further observe that models with comparable detection accuracy can exhibit markedly different computational costs, and that low-bit quantization largely preserves detection performance in the evaluated configurations. Finally, we examine detection robustness under structural, semantic, and label noise at different perturbation levels. These findings provide empirical insights into the performance, efficiency, and robustness of LLM-based log anomaly detection, highlighting practical considerations beyond conventional accuracy-oriented evaluation.