Dynamic network penetration testing improved by hierarchical reinforcement learning
DYNAS: A Dynamic Network Penetration Testing Framework Using Deep Hierarchical Reinforcement Learning
Cryptography and SecurityMachine LearningNetworking and Internet Architecture
Summary
Testing computer networks for vulnerabilities is harder when networks are large and changing. The authors built a tool called DYNAS that simulates realistic network defenses and challenges. They created a smart agent, Pen-DHRL, that learns to break into these networks by setting step-by-step goals using information from intrusion detection systems. In tests, this approach outperformed other methods, especially when defenses were active during training. It shows promise for more realistic and effective automated network testing.
What this means in practice
- •For network security engineers: Evaluate and improve automated penetration testing tools by using DYNAS to simulate realistic network defenses and uncertainties during training and testing.
- •For cyber defense teams: Train and test intrusion detection systems under dynamic attack scenarios generated by DYNAS to better assess defense robustness against adaptive attacks.
Authors
Md. Rakibul Hassan, Majedul Islam, Md. Ashraful Alam, Muhammad Iqbal Hossain, Swakkhar Shatabda
Abstract
As network scale and uncertainty complicate automated penetration testing, reinforcement learning (RL) environments must support controlled evaluation under defensive and non-stationary conditions. We present DYNAS (Dynamic Network Attack Simulator), which combines parameterized large-scale scenario generation with stateful intrusion detection responses, explicit scan false-positive and false-negative noise, temporary service unavailability, and duration-based action timeouts. We further introduce Pen-DHRL, a recurrent manager--worker policy that selects persistent subgoals using IDS-conditioned global context. We compare Pen-DHRL with GNN, Invariant, and Attention baselines across four scenario variants and multiple training regimes. Under IDS-disabled training followed by IDS-enabled evaluation, Pen-DHRL obtains approximately 64\% and 24\% higher reward per step on Bridge and Dynamic, respectively, and approximately 26\% and 30\% more captured hosts on VarA and VarB than the next-best baseline. With all defensive dynamics active from the beginning of training, Pen-DHRL achieves positive mean episode returns in all four scenarios, whereas the flat baselines produce negative mean returns in all 12 architecture--scenario combinations; staged curriculum training produces positive mean returns across all 16 evaluated combinations.