Unsupervised anomaly detection methods vary widely by log data and preprocessing

Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

Software Engineering

Summary

Detecting unusual problems in software logs helps keep systems running smoothly. The authors studied different ways to detect these problems without needing labeled examples. They found that performance varies a lot depending on the dataset, how the logs are prepared, and which detection method is used. For some datasets, parsing logs into events helps, while for others it does not. Using clean, anomaly-free data for training can improve detection but requires extra effort.

What this means in practice

  • For software reliability engineers: Choose anomaly detection techniques and preprocessing steps tailored to specific software logs to better identify failures and faults without labeled training data.
  • For it operations teams: Implement faster anomaly detection pipelines by using word-based log representations and avoiding computationally expensive models like IsolationForest for quicker fault identification.

Authors

Jesse Nyyssölä, Mika Mäntylä, Yuqing Wang

Abstract

Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data.Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. For fast execution of the pipeline, use word representation and avoid IsolationForest model.