RankShift: In-Database Detection and Explanation of Categorical Shifts
Machine Learning
Summary
The authors present RankShift, a method to detect unusual changes in categories within log data, even when the total number of events stays the same. It works by comparing category distributions over time using a statistical score and highlights which categories are causing the change. They tested RankShift on three datasets and found it performs similarly or better than advanced models like autoencoders, especially in spotting subtle shifts in rare event categories. Unlike autoencoders, RankShift needs no training and uses much less storage.
Authors
Omair Shafi Ahmed
Abstract
A login service can receive its usual number of failed sign-ins while one source grows from 2% to 30% of them. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable. These events change which categories are active without changing how many events occur. RankShift detects such changes inside the analytical database that stores the data. It compares each window's category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change. The same query returns the score, calibrated alert, and largest increasing contributions. We evaluate RankShift on HDFS, BGL, and Thunderbird. It matches the count-vector autoencoder within 0.001 AUROC on HDFS (0.999 versus 1.000) and leads on Thunderbird (0.983 versus 0.949). In a controlled fixed-volume experiment, RankShift detects rare-category shifts that are invisible to event-count monitoring, reaching 0.787 AUROC compared with 0.771 for the autoencoder. Across all three corpora, observed false-alarm rates track the requested operating levels. RankShift requires no model training or inference service, and the autoencoders deployed state is 137x larger.