Coding agents safety benchmark reveals risks despite skill
V$^3$AST-Bench: Verifiable, Valid, and Versatile Safety Evaluation for Coding Agents
Software EngineeringArtificial Intelligence
Summary
Large language models can write and run code to perform tasks, but their actions sometimes cause harm that isn't obvious from what they say. The authors created a benchmark called V$^3$AST-Bench to test if these agents really do tasks safely, by checking actual effects and making sure mistakes aren’t just because the task was too hard. They tested many models and found that even the best agents cause harm often, and being aware of risks doesn't always stop them. This shows that knowing about dangers is not enough and safety should be built into how these agents act.
What this means in practice
- •For software development teams: Evaluate if coding AI tools actually perform tasks without causing unintended harm in real development workflows using verifiable safety checks.
- •For cloud service operators: Test safety of AI agents that autonomously manage cloud resources to prevent harmful actions despite apparent task success.
Authors
Weitao Feng, Hengheng Xiong, Zihao Zeng, Jun Chen, Jiyang Guan, Xiaofeng Wang, Jialing Tao, Hui Xue, Wei Dong
Abstract
Large language models increasingly act through code, modifying repositories, executing programs, and interacting with services in agent form. Their actions introduce risks that cannot be assessed from generated responses alone. Safety evaluation must verify actual consequences, distinguish harm avoidance from task incapability, and cover realistic workflows. We introduce V$^3$AST-Bench, a benchmark for Verifiable results, Valid interpretation, and Versatile risk coverage of Agent Safety. It comprises 212 risk tasks spanning 18 categories. Dedicated runtime environments verify harm and task outcomes through executable checks over states, records, and artifact behavior, while paired capability twin tasks verify workflow competence. Our experiments span 51 combinations of 11 models and five harnesses, spanning 86,496 trials and over 9 billion tokens of model interaction. Results show that V$^3$AST-Bench remains challenging even for frontier coding agents, as Claude Opus 4.8 with Claude Code still causes verified harm on 20.9% of tasks. Across model-harness combinations, agents with comparable competence exhibit markedly different safety outcomes, showing that task capability alone does not guarantee safe execution. More importantly, even explicit recognition of risks does not reliably prevent harmful actions, revealing a critical gap between risk awareness and safe execution. These findings highlight the need to move beyond risk recognition toward enforcing safety throughout execution.