Policy transfer enables cyber agents to work across simulators and emulators
Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents
Cryptography and SecurityMachine Learning
Summary
Training cyber attack agents in one simulation environment often means they don’t work well in others because these environments are very different. The authors treat transferring policies between simulators and to real cyber environments as a single challenge of matching states and actions. They created a system that lets an agent trained in one environment operate in another without needing to learn again. Their tests show that this approach can keep performance when environments are similar and still perform reasonably well even when they differ. This could help build cyber defense tools that adapt more easily to different setups.
What this means in practice
- •For cybersecurity operations teams: Use transferred attack-defense policies across different simulation tools to evaluate and improve network security without retraining agents.
- •For virtual environment developers: Integrate policy translation frameworks to facilitate behavioral consistency of agents when deployed across diverse cyber simulation platforms and emulators.
Authors
Sabrina Saika, Yinuo Du, Aritran Piplai
Abstract
Cyber attack agents are typically trained and evaluated within a single simulator, making it unclear whether learned policies transfer beyond the environments in which they were developed. This limitation hinders both deployment and fair comparison, as cyber simulators differ substantially in their state representations, observation models, and action spaces. In this paper, we study policy transfer across cyber environments and argue that simulator-to-simulator and simulator-to-real transfer can be viewed as instances of the same underlying alignment problem. We propose a framework that separates state alignment from action translation, enabling a policy trained in one environment to operate in another without retraining. We evaluate transfer across four cyber platforms, CyberBattleSim, NetSecGame, CyberWheel, and NASim, including emulated deployments in NASim. Our experiments show that zero-shot transfer is feasible, fully preserving source-policy performance in closely aligned environments and achieving 45.2% win rates when transferring policies whose source performance is 60.5%. In emulated virtual machine environments, transferred policies exhibit a Jensen-Shannon divergence of 0.085 from native policies, indicating strong behavioral similarity. Code and benchmarks are available at: https://anonymous.4open.science/r/RL-Transfer-between-env-4F47/.