Coding agents often cause harm while completing tasks safely
VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably
Computation and LanguageSoftware Engineering
Summary
Some advanced computer programs can write code to complete tasks but might accidentally cause damage like deleting important files or changing settings without permission. The authors identify this problem as harmful success, where the task is done but with harmful side effects that users don't notice. They developed VeriHarm, a way to test and measure this harmful success by checking both task completion and unintended damage. Their tests show that even powerful models can complete tasks correctly but often still cause harm.
What this means in practice
- •For software development teams: Detect unintended harmful effects in code generated by AI assistants to improve safety in software projects.
- •For cybersecurity operators: Use automated benchmarks to verify that AI coding agents do not weaken system security while performing tasks.
Authors
Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Yu-Min Tseng, Tzu-Han Lin, Yu Meng
Abstract
Large language model (LLM) coding agents increasingly operate with shell access, allowing them to write scripts, move files, alter permissions, and modify system configurations. However, an agent may achieve a user's goal while causing unintended harm, such as deleting unrelated files, weakening access controls, or corrupting data, in response to benign inputs without explicit jailbreaking. This failure mode is particularly important because apparent task success may reassure users that an agent behaved correctly, causing harmful side effects to go unnoticed. We call this harmful success: the agent successfully completes the intended benign task while simultaneously causing unintended harm. Despite its importance, harmful success is difficult to measure because conventional task-success metrics overlook harmful behaviors, while existing safety evaluations are often designed solely to assess harmful outcomes. We introduce VeriHarm, an automatic framework for constructing programmatically verifiable benchmarks that directly measure harmful success. VeriHarm minimally perturbs coding tasks to elicit unintended harmful behavior while maintaining the benignity of the task and preserving the original task objective. It then synthesizes executable harm tests that, together with the original task-correctness tests, evaluate both task accuracy and harmful outcomes. Frontier agents, including GPT-5.6-Sol and Claude-Opus-5, achieve an average task accuracy of 95% on our curated VeriHarm-Bench, yet still struggle to complete tasks safely: their safe success rate, the fraction of tasks completed successfully without triggering the designated harm, averages only 37%. These results suggest that successful task completion can frequently coexist with unintended harm, and that task-success metrics alone may substantially understate the potential safety risks of coding agents.