Cyberarena benchmarks ai agents in live cyber attack and defense battles
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Cryptography and SecurityArtificial IntelligenceMultiagent Systems
Summary
Cybersecurity involves attackers trying to break into systems and defenders trying to stop them. The authors point out that most current tests check AI agents only against fixed problems, missing how real attacks and defenses change together over time. They created CyberArena, where two AI agents fight each other live, both attacking and defending at the same time in a shared environment that changes each round. This setup better shows how well AI can handle the fast and complex nature of cybersecurity battles.
What this means in practice
- •For cybersecurity operations teams: Test and compare AI tools in realistic, ongoing attack-and-defense scenarios to find which perform best at protecting live systems.
- •For software security developers: Use CyberArena to simulate multi-round exploit and patch cycles, helping develop better defensive methods and automated vulnerability fixes.
Authors
Bangya Liu, Jianhong Tu, Yueyang Pan, Hanhong Zhao, Zhe Ye, Hongwei Li, Saastha Vasan, Katherine Lambert, Junmin Zhu, Hao Wang, Jienan Chen, Yunze Xiao, Minglai Yang, Zihan Wang, Hairong Yin, Hao Wang, Zhiwen Fan, Tianneng Shi, Zhun Wang, Chenguang Wang, Dawn Song
Abstract
AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static benchmarks quickly saturate and miss an agent's ability to attack and defend simultaneously under live pressure. To address this gap, we present CyberArena, a dynamic, symmetric adversarial benchmark built around multi-round attack-and-defense combat. In isolated sandboxes, two agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an agent's speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain the benchmark's longevity, an automated agentic pipeline continuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce a dynamical model that maps match dynamics into unified, multidimensional scores across offensive, defensive, and domain-specific capabilities. Evaluating 12 frontier LLM agents across 40 challenges in 466 matches, CyberArena separates model capabilities where single-agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a benchmark, CyberArena serves as an adaptable, scalable experimental platform for future research on cybersecurity agents.