Coding agents struggle to build kernel exploits without examples
Evaluating Coding Agents on Kernel Exploit Generation
Artificial IntelligenceCryptography and Security
Summary
Finding software bugs does not guarantee the ability to make those bugs into useful exploits. The authors created KEX-bench, a test suite to see if smart coding agents can actually generate small building blocks of exploits against real operating system kernels like Linux and Windows. They found agents can often cause crashes but usually fail to craft these critical exploit parts without example exploits given. With example exploits, agents do much better, showing a gap in current capabilities. The benchmark is publicly available for others to try and improve.
What this means in practice
- •For security testing teams: Evaluate automated coding tools on their ability to generate exploit components against real Linux and Windows kernel vulnerabilities.
- •For software quality assurance teams: Use the benchmark to measure and improve bug exploitation capabilities of AI-assisted coding before deployment.
Authors
Junyoung Jang, Gwanhyun Lee, Hwiwon Lee, Kyuheon Kim, Jongseong Kim, Jinho Jung, Lingming Zhang
Abstract
Coding agents now find real vulnerabilities in production software. However, bug discovery results do not measure whether agents can construct exploit primitives. We introduce KEX-bench, a benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels. KEX-bench contains 45 task instances across 40 Linux and Windows CVEs, covering kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write. Each task runs in an isolated virtual machine, exposes controlled tools, and uses a deterministic verifier to check primitive-specific success. We evaluate state-of-the-art coding agents paired with frontier and open-weight models under fixed tool-call budgets. Without a reference proof of concept (PoC), the strongest configuration solves 1 of 20 Windows tasks (5.0%) and 14 of 25 Linux tasks (56.0%). With a reference PoC, the strongest configuration solves 31 of 45 tasks (68.9%). This highlights the gap where agents reach kernel crashes but fail to shape kernel state into exploit primitives. We release KEX-bench for reproducible research on AI-assisted exploitation at https://kex-bench.github.io.