Host intrusion detection improved by reorganizing system call sequences
A Resource-Aware Behavior Reconstruction and Hierarchical Semantic Learning Framework for Host Intrusion Detection
Cryptography and Security
Summary
Detecting malicious software on computers involves tracking how programs talk to the system, often through system calls. The authors found that using raw recorded sequences can mix up behaviors from different programs, making it hard to spot attacks. They developed a method called ReSHID that reorganizes these sequences based on identity and relationships, then uses a graph-based approach to understand complex attacks. Their method improved detection accuracy and reduced the amount of data needed to analyze.
What this means in practice
- •For cloud security teams: Enhance detection of attacks on cloud servers by restructuring system call data to reveal clearer attack patterns.
- •For enterprise it security teams: Improve host-based intrusion detection performance using lightweight models informed by subject behavior graphs for better threat identification.
Authors
Youli Tao, Rui Tang, Hao Ren, Chengsheng Zhou, Dengzhe Wang, Shuyu Jiang, Xingshu Chen
Abstract
System calls (syscalls) record key interactions between running programs and the operating system kernel, providing fine-grained and minimally intrusive data for host-based intrusion detection systems (HIDS) deployed in cloud and other modern computing environments. However, existing methods often model syscalls in their original execution order, where sequences from different processes are interleaved, making informative patterns difficult to extract and raising two questions: whether raw syscall sequences can be reorganized in a way that yields more discriminative representations, and how complex attack patterns can be effectively learned from the reorganized sequences. We propose ReSHID, a resource-aware behavior reconstruction and hierarchical semantic learning framework for host intrusion detection. It reconstructs semantically continuous sequences by leveraging syscall semantic invariants to cast subject identity and relationship resolution across PID namespaces as a bipartite matching problem and tracking file descriptor (FD) lifecycles to associate descriptors referring to the same resource. Additionally, features extracted from these sequences are organized into a lightweight subject behavior graph incorporating inter-subject relationships, where GATv2 captures key coordination patterns to model complex attacks involving multiple subjects. Experimental results show that sequence reconstruction combined with the detection method can improve HIDS performance. Even with a lightweight linear classifier, the proposed method achieves the best results among all compared methods in terms of F1-score (98.64%), ROC-AUC (99.80%), and PR-AUC (98.10%), while reducing the number of n-gram features by approximately 75.2% and 44.1% compared with the raw sequences and MGFE, respectively.