Guardrails need to allow authorized actions while blocking unauthorized ones
Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization
Artificial IntelligenceSoftware Engineering
Summary
When AI systems perform tasks like writing code or moving money, it’s important they only do what users are allowed to do. The authors studied how safety guards (called guardrails) decide what is allowed. They created a test to check if these guardrails block unauthorized actions but still allow the same actions when permitted. They found many errors come from mistakes in checking, and fixing those can change which allowed actions get through. The study shows guardrails must be evaluated on both stopping bad actions and letting good ones happen.
What this means in practice
- •For security engineers: Test guardrails in AI systems to ensure they correctly block unauthorized actions and allow authorized ones.
- •For financial software developers: Validate authorization checks in AI tools that manage transactions to reduce errors in permission enforcement.
Authors
Huihao Jing, Shaojin Chen, Wenbin Hu, Yuxuan Liu, Haochen Shi, Zhongwei Xie, Haoran Li, Yangqiu Song
Abstract
As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem. Guardrails are often placed in front of these actions, yet few evaluations ask whether their decisions follow authorization: blocking unauthorized actions while still allowing the same actions under valid permission. We introduce \bench{}, a benchmark that contrasts authorization evidence for the same operation in its language track. It combines expert-annotated natural-language scenarios with executed and verified operations, plus externally sourced native-policy requests. \bench{} checks each decision for correctness, usability and resulting effect, and tracks each request across guardrail changes. Our experiments show that many blocked operations stem from failed reviews, not correct denials. Repairing the output format restores usable decisions but changes which authorized requests are honored, and adding a second reviewer removes some errors while rejecting valid authorization. Guardrails should therefore be evaluated on both the unauthorized actions they stop and the authorized actions they let through.