Summary
Detecting attacks on Internet-connected devices using machine learning often looks very accurate, but this paper finds that many models learn tricks specific to the test data instead of real attack behaviors. The authors tested several models and feature types on a large IoT network dataset, showing that models can take advantage of time stamps and repeated session info rather than generalizing well. Some attacks are easier to detect than others, and careless testing methods can make performance look better than it really is. They suggest a checklist to better evaluate intrusion detection systems realistically.
What this means in practice
- •For cybersecurity teams: Improve evaluation of IoT intrusion detection models by avoiding misleading dataset shortcuts and ensuring realistic testing protocols.
- •For network operations centers: Better assess detection performance on IoT traffic by selecting features and data splits that prevent overestimating attack detectability.
Abstract
Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models learn generalizable attack behavior or exploit spurious dataset shortcuts- such as static testbed IP/MAC addresses and chronological recording artifacts-remains an important question. We evaluate the CyberFlowIoT-GICAP benchmark, containing 3,617,388 flow records across 126 PCAP sessions with 849,395 benign flows. Four learning paradigms are evaluated across four feature configurations using PCAP-disjoint splits; LightGBM is additionally evaluated using conventional random-flow splitting. When only statistical flow behavior is used (Fbehav), LightGBM (92.58% +/- 8.18%), Random Forest (92.59% +/- 8.18%), and Deep MLP (92.55% +/- 8.18%) achieve nearly identical Macro-F1, indicating that performance is constrained by feature representation rather than model complexity. With raw timestamps (Ftstamp), tree-based models reach 99.28% Macro-F1, while the linear model remains at 90.62%, showing that nonlinear models can exploit dataset-specific temporal structure. Attack detectability is highly asymmetric: high-rate and active attacks maintain >99.8% recall from flow behavior alone in nonlinear models, whereas the DNS Beaconing drops from 27.78% to 0.00% recall when contextual features are removed. Conventional random-flow splitting increases attack recall by up to 14.00%, highlighting the effect of placing flows from the same sessions in both training and test sets. We conclude with a 4-point protocol checklist for realistic IoT NIDS evaluation.