Adaptive methods align autonomous cyber defense to human goals
Alignment of Autonomous Cyber Defense Agents via Adaptive Policy Shaping
Cryptography and Security
Summary
Cyber attacks are becoming more complex and quicker, so computers that defend networks on their own are needed. These computer defenders learn how to act, but sometimes their actions cause other problems like slowing down services or breaking rules. The authors studied a way to adjust these computer defenders' behavior while they work, without needing to train them again from scratch. Their approach can balance protecting the network and keeping it running smoothly based on changing priorities in an attack. They tested their ideas in a simulated environment and showed it helps the defenders better follow human goals without losing effectiveness.
What this means in practice
- •For cybersecurity operations teams: Tune autonomous cyber defense agents in real time to better meet changing security and operational priorities during attacks.
- •For network administrators: Maintain service availability and security by adapting defense agent policies without retraining when operational goals shift.
Authors
Dena Mujtaba, Bharadwaj Ravichandran, Brian Hu, Arslan Basharat
Abstract
Cybersecurity operators increasingly encounter sophisticated and fast-paced attacks, making autonomous cyber defense (ACD) a critical approach for supporting timely and effective decision-making. Reinforcement learning (RL)-based ACD agents have the ability to autonomously observe network states and take actions to defend and recover system assets, making them well suited for ACD. However, these agents often face critical trade-offs, such as improving network security while inadvertently degrading service availability or violating other operational priorities such as confidentiality and integrity, raising the core challenge of aligning agent behavior with human-specified goals and intent. In this work, we study the problem of aligning RL-based ACD agents with human-specified operational objectives without requiring costly retraining. We formalize alignment for ACD as a policy shaping problem, where a pretrained RL decision-making policy can be adjusted at test time using predicted attributes over agent actions. We also propose an adaptive approach that dynamically controls the alignment--performance trade-off as operational priorities shift during an ongoing cyber attack. We evaluate our method in a simulated cyber defense environment, CybORG, analyzing performance across multiple alignment objectives using the Confidentiality, Integrity, and Availability (CIA) information security triad. Our results show that adaptive test-time policy shaping can significantly improve alignment while preserving core cyber defense performance, offering a practical and scalable alternative to training-time alignment methods in RL. Our code is publicly available at https://github.com/ITM-Kitware/alignment-cage-challenge-2.