Hybrid system combining Suricata and Random Forest struggles in real networks
Passive Hybrid Network-Based Intrusion Detection System (Hybrid-NIDS) Combining Suricata and Random Forest
Cryptography and Security
Summary
Detecting cyberattacks using computer programs often relies on testing with standard datasets before using them in real networks. The authors built a system that combines a popular tool called Suricata with a machine learning method called Random Forest to spot attacks. While their system performed well on benchmark data, it missed almost all attacks when tested on real network traffic. This means success in controlled tests does not guarantee effectiveness in real-world settings.
What this means in practice
- •For network security teams: Use the prototype framework to compare Suricata and machine learning for passive attack detection and explore domain shift effects.
- •For cybersecurity engineers: Investigate limitations of ML-based intrusion detection in lab and real network traffic to improve alert handling systems.
Authors
Quoc-Cuong Tang, Hoang-Lam Huynh, Van-Tri Phan, Khuong Nguyen-An
Abstract
This paper evaluates a passive Hybrid Network-based Intrusion Detection System (Hybrid-NIDS) prototype that combines Suricata with Random Forest flow classification and centralized ELK-based alert handling. The study explicitly separates benchmark evaluation from PCAP/live operational validation and controls exact feature-duplicate leakage using feature hashing and group-aware splitting. From 2,540,047 UNSW-NB15 records, 453 conflicting-label hash groups containing 1,879 rows were removed; the resulting Development and Hold-out sets have zero exact feature-hash overlap. RF-41 achieved F1 = 0.971360 and ROC-AUC = 0.999671, while the NFStream-compatible RF-21 achieved F1 = 0.970148 on the same prepared hold-out boundary. However, operational validation revealed substantial benchmark-to-deployment domain shift: on a labeled laboratory PCAP, RF-21 and the strictly correlated branch achieved recall of only 0.0095, and RF-21 produced no alerts in five additional 60-second attack sessions. An unlabeled normal-traffic test produced 439 alerts from 2,375 flows; this value is reported only as an alert ratio and is not interpreted as a false-positive rate. These results show that strong performance on a public benchmark does not directly translate into operational effectiveness. Accordingly, the current Hybrid-NIDS should be interpreted as a passive prototype and evaluation framework, and the reported experiments do not demonstrate that Suricata-Random Forest correlation provides better operational detection than Suricata alone.