Bounded auks metric improves anomaly detection evaluation accuracy
Bounded AUKS: A Novel Evaluation Metric for Anomaly Detection via Bounded Area Under the Kolmogorov--Smirnov Curve
Computer Vision and Pattern Recognition
Summary
Many current ways to measure how well computers find unusual spots in images can be misleading because they don't consider important details about decision thresholds. The authors propose a new measurement called Bounded AUKS that focuses on the meaningful range of detection scores, avoiding regions that give false confidence or hide real problems. This new metric better shows how well the system separates normal from unusual parts. They tested it on popular datasets and models, finding it gives a clearer, more honest evaluation than older methods.
What this means in practice
- •For machine learning engineers: Evaluate pixel-level anomaly detection models more accurately by using Bounded AUKS to assess separability in score distributions within practical threshold ranges.
- •For quality control teams: Better interpret automated inspection results to identify manufacturing defects by relying on evaluation metrics that reflect realistic detection operating points.
Authors
Yanshuo Zhang, Juan Du
Abstract
Recent literature widely used global metrics such as AUROC and AUPRO to quantify the detection performance. AUROC and AUPRO scores are approaching 100% in some public datasets such as MVTec AD dataset and VisA dataset, which misleadingly suggests that the pixel-level anomaly detection task has already been solved. Threshold-independent metrics represented by AUROC inherently ignore the threshold selection process, which is inevitable in practical applications. Although AUPRO restricts the False Positive rates (FPRs) to an acceptable range, extremely low FPRs can lead to increased false negative rates (FNRs). We argue that there is a lack of metrics capable of quantifying the separation between the anomaly score distributions of normal and anomalous samples, which hinders advancements in anomaly detection. In response, we propose the Bounded Area Under the Kolmogorov--Smirnov curve (Bounded AUKS), a partial integral metric that integrates the difference between TPR and FPR within a data-driven bounded range. Specifically, we introduce AUKS from KS integral as a direct measure of separability between the anomaly score distributions. The bounded range is determined only by score distributions of normal test samples. Lower bound excludes the uninformative high-FPR region. Upper bound avoids evaluation in extreme regions that aggressively push FPR towards zero, which result in a very low recall. Our experiments on 9 models across MVTec AD and VisA datasets show that Bounded AUKS can address the limitations of existing metrics and has a better performance in evaluating unsupervised anomaly detection models.