Static methods outperform streaming ones for time series anomalies
In a Streaming World, Should You Stand Still? A Comprehensive Benchmark of Anomaly Detection in Streams
Machine LearningArtificial Intelligence
Summary
Detecting unusual changes in data that comes in over time is important but challenging. The authors found that methods which analyze a fixed batch of past data usually spot anomalies more accurately than those that update their model continuously as new data arrives. They created a new dataset to test when updating models might help and found that most streaming methods don't perform as well in realistic scenarios. This suggests that the common belief in the superiority of streaming updates for anomaly detection should be reconsidered.
What this means in practice
- •For data engineering teams: Choose static batch-trained anomaly detectors over streaming updates for more accurate detection in real operating environments.
- •For iot device operators: Monitor sensor data using static anomaly detection methods to reduce false alarms and improve reliability in non-stationary environments.
Authors
Magali Parrino, Antoine Ajenjo, Emmanuel Remy, Pierre Stephan, Pierre Senellart, Paul Boniol
Abstract
Time series anomaly detection (TSAD) is increasingly deployed in streaming settings, where data arrive sequentially and may exhibit non-stationarity. As a result, several works from the recent literature propose streaming anomaly detection methods that rely on incremental updates to adapt over time. However, most of these approaches originate from the streaming outlier detection literature and largely ignore core characteristics of time series anomalies. Moreover, their empirical evaluation is typically conducted on synthetic or small-scale benchmarks with limited diversity, making it unclear whether streaming methods are truly advantageous in realistic TSAD scenarios. In this work, we carry out the first large-scale experimental study comparing streaming and static TSAD methods under a unified streaming evaluation benchmark. We consider a realistic setting in which an initial batch of data is available for model training, followed by online evaluation of both detection accuracy and computational efficiency. In addition, we propose a distribution-drift dataset of real time series, called TSB- drift, to isolate scenarios where streaming updates are theoretically justified. Our results show that, contrary to common assumptions, static TSAD methods significantly outperform streaming approaches in most streaming settings. Such finding highlights a critical gap between the design of existing streaming methods and the requirements of modern TSAD, and calls for a rethinking of how streaming capabilities should be integrated into TSAD.