Provenance based intrusion detectors need careful score audits
Where the Numbers Come From: Auditing Evaluation in Provenance-Based Intrusion Detection
Cryptography and Security
Summary
Provenance-based intrusion detection systems generate scores to decide if alarms indicate an attack, but these scores can be misleading if their calculation isn't clearly understood. The authors examined nine such detectors and found that some reported precision numbers don’t match the actual attack labels in their alarms. They also uncovered technical issues like buffer reuse errors affecting results and showed that choosing test labels carefully can skew performance numbers. Their work highlights the importance of transparent evaluation and reporting to fairly compare these detectors.
What this means in practice
- •For cybersecurity engineers: Audit and verify provenance based intrusion detector scores to avoid misleading alarm precision claims.
- •For security software developers: Fix buffer reuse and label selection issues in provenance encoders to improve attack detection accuracy.
Authors
Jihwan Moon, Gunhee Kim, Myeongjang Pyeon
Abstract
Reproducing a provenance-based intrusion detector's score does not establish what that score says about its emitted alarms or the information its encoder uses. We audit nine released implementations, execute four detectors using their own code, and isolate three measurement effects. First, a fixed-alert comparison separates label choice from neighbourhood credit: ThreaTrace reports precision 0.938 with neighbourhood credit, although only seven of its 994 alarms carry its own attack label. Second, removing test-label checkpoint selection lowers attack detection precision (ADP) by 0.16 to 0.33 across four forty-member word2vec configurations without changing detector order. Third, a buffer-reuse defect gives a linear encoder unintended degree-dependent inputs. Correcting it lowers type-only ADP in every identical-input initialization pair on two hosts, while historical word2vec effects depend on the host. These findings qualify claims from the inspected implementations about alarm precision, performance magnitude and static-attribute sufficiency. STRICT connects them to six checkable reporting requirements. Because the comparisons condition on benchmark targets, they neither validate those labels nor establish a universal detector ranking.