PairAudit improves error detection in intrusion reviews with graph tokens
PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift
Machine LearningCryptography and Security
Summary
Intrusion detection systems can confidently make mistakes when encountering attacks they haven't seen before. The authors introduce PairAudit, a tool that helps human reviewers find these hidden errors by looking at unusual patterns in how predictions relate to each other. Instead of only focusing on how uncertain a detector is, PairAudit uses relationships between data points to suggest which cases should be reviewed. This approach lets reviewers fix more errors without needing to retrain the original detection system.
What this means in practice
- •For security analysts: Improve prioritization of manual reviews to detect hidden intrusion errors missed by automated systems.
- •For network operations teams: Guide efficient incident investigation by highlighting unusual prediction patterns indicating potential unseen attacks.
Authors
Jiran Tao, Binyan Jiang
Abstract
Intrusion detectors can confidently misclassify attacks that were not seen during training. Human review can correct these errors, but only a limited number of cases can be checked. Uncertainty-based review may overlook confident errors, while anomaly scores alone do not show whether changing the review plan will correct more errors. We introduce PairAudit to find overlooked errors and improve review under a fixed budget. Its graph tokens capture prediction patterns across connected nodes. Rather than building another predictor through feature aggregation, PairAudit uses unusual relational patterns to uncover potential errors in existing predictions. Human feedback then helps decide whether these findings justify changing review priorities. Experiments across security tasks show that PairAudit corrects more errors on average than uncertainty-based review, including more errors on unseen attacks. These gains account for all review costs and do not require retraining the detector.