Multimodal intrusion detection performs differently with recording-aware evaluation
Recording-Aware Evaluation of Multimodal OT Intrusion Detection
Cryptography and SecurityMachine Learning
Summary
Intrusion detection systems for industrial control networks are usually tested on separate recordings for normal and attack data, which can cause misleadingly high scores due to differences in recordings rather than true detection ability. The authors propose a new way to evaluate these systems that compares attack data to normal data from the same recording, better reflecting real detection performance. They found that physical sensor data become more important under this evaluation, and that common performance metrics can overstate detection effectiveness when alarms last a long time. This suggests that current benchmarks should be interpreted with more context.
What this means in practice
- •For industrial cybersecurity teams: Evaluate intrusion detection systems in industrial settings using same-recording data to obtain more accurate detection performance measures.
- •For critical infrastructure operators: Use improved evaluation methods to choose IDS solutions that reliably detect attacks without inflated alarms caused by recording differences.
Authors
Mustafa Umut Ozbek, Khalil El-Khatib, Li Yang
Abstract
Machine-learning-based intrusion detection systems (IDSs) for operational technology (OT) are commonly evaluated on benchmark datasets in which normal and attack data are collected in separate recordings. In such settings, high performance may reflect recording-associated differences as well as attack behavior. We introduce a recording-aware evaluation procedure that compares attack windows with background from the same recording, uses prior data for model fitting and calibration, tests temporal alignment against randomized event schedules, and reports event coverage together with alarm burden. On the ProvICS multimodal dataset, cyber modalities strongly distinguish the benign and attack recordings before the first labeled attack. Same-recording evaluation reverses the apparent modality ranking across all studied window sizes: the physical view becomes the strongest individual modality, while cyber-only fusion shows the largest difference between cross-recording and same-recording AUC, reaching 0.54. Operational analysis further shows that permissive event-overlap metrics can overstate detection when a small number of long alarm episodes cover multiple actions. An external analysis on HAI shows smaller and reference-dependent recording sensitivity. These findings show that cross-recording benchmark performance should be interpreted alongside same-recording, forward, temporal, and operational evaluations.