System One models show limits in automated agent security decisions
Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
Summary
Detecting harmful inputs to AI agents is hard, especially when models mistakenly treat attacks as safe. The authors tested several security judging systems and found that even if these models look accurate overall, they can confidently miss certain types of attacks. Using strict rules to block or allow input leads to rejecting many inputs automatically and still missing some attacks on new data. Adding a second model to catch errors helps but also blocks more safe inputs and repeats some mistakes. The authors emphasize checking model accuracy, confidence, and decision rules together to better secure AI agents.
What this means in practice
- •For security operation teams: Improve automated screening policies to better balance missed attacks and false blocks in AI-powered threat detection.
- •For software engineers: Develop multi-model security layers that reduce undetected attacks but manage increased false rejections in user inputs.
Authors
Yixuan Liu
Abstract
Model-based judges support agent security by detecting prompt injections, assessing interaction risks, and screening harmful requests. System One models expose typed decisions with probabilities that software can use to allow, block, or escalate inputs, but whether these probabilities support reliable automated security decisions remains unclear. We evaluate Jev, Laya, Decider, and Bespoke Nimble against specialized classifiers and language-model judges, examining decision accuracy, probability calibration, and selective automation. We identify three main findings. (1) Strong overall performance and favorable average calibration can hide systematic failures on particular attack groups, including attacks that models confidently classify as safe. (2) Under strict limits on missed attacks, the evaluated policies allow few inputs automatically, and choosing separate allow and block thresholds increases automation mainly by blocking more inputs. Even policies that meet error limits during validation can exceed them on unseen inputs. (3) A second judge can detect some missed attacks, but it may also reject more benign inputs and repeat the first model's high-confidence errors. These findings show that model accuracy, probability calibration, and the behavior of the resulting decision policy must be evaluated together.