Tool using language models risk unauthorized side effects in tasks
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents
Artificial IntelligenceComputation and Language
Summary
Language models that use tools to help complete tasks are usually judged successful if they finish the job. This paper shows that sometimes these models complete tasks but also make unintended or unauthorized changes, called collateral damage. The authors created a benchmark with many tasks involving email, calendars, and tickets to measure these side effects. They found that some types of tasks and states cause more of these errors, and different models vary greatly in this risk. Their findings suggest just finishing a task isn’t enough to say these models are safe.
What this means in practice
- •For software developers: Evaluate AI assistants for unauthorized actions beyond task completion using the SABench benchmark and oracle.
- •For enterprise automation teams: Detect and reduce unsafe side effects in AI-driven email and scheduling automations by testing with SABench.
Authors
Jiaju Huang, Shaobin Chen, Xinglong Liang, Xinyu Ma, Hao Yang, Wei Ke, Tao Tan, Yue Sun
Abstract
Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can complete the requested work while also taking an unauthorized side-effecting action, such as writing to the wrong object, expanding the scope of the request, or contacting an unintended recipient. We call this failure mode collateral damage (CD). We introduce SABench (Side-effect-Aware Benchmark), a benchmark of 201 email, calendar, and ticket tasks. Each task crosses one of four state structures (independent, dependent, externally reachable, cross-tool) with one of four trap families (ambiguity, mis-targeting, cascade harm, overreach), and each task has a deterministic trace-based oracle for both goal satisfaction and authorization compliance. Across 2,211 model-task pairs from 11 frontier LLMs spanning major commercial and open model families, 15.2% of all runs incur CD, with risk concentrated in ambiguity tasks (38.8% violation rate) while non-ambiguous traps average 8.4% CD. Among goal-satisfied runs, 12.8% still produce unauthorized writes. Cross-tool and dependent states have about twice the CD rate of independent states. CD rates vary by 4.7x across models at comparable goal rates, which suggests that task success alone is a poor proxy for safe tool use. The SABench task corpus is released at https://huggingface.co/datasets/Phaedrus/sabench; the traces, oracle, code, and analysis pipeline are released at https://github.com/hjj194/sabench-artifact.