Guard models miss unfulfilled safety actions in AI agents

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

Artificial Intelligence

Summary

Many AI systems are designed to avoid doing unsafe things, but this paper shows that just avoiding bad actions is not enough to keep them safe. The researchers found that AI agents often fail to take important safe actions they are supposed to do, which can be risky. They created a new test called ObligationBench to measure how well AI systems recognize these required but missing actions. Their new method, ObligationGuard, performs better than existing systems at spotting these missed safety steps, suggesting future AI safety tools should look for both bad actions and missing good actions.

What this means in practice

  • For ai system developers: Improve safety checks in AI agents by detecting required safety actions they fail to perform, beyond just forbidding unsafe actions.
  • For automation platform engineers: Enhance automation reliability by integrating obligation detection to ensure critical safety tasks are not missed during execution.

Authors

Youwei Feng, Yitong Zhang, Yuetong Liu, Jia Li

Abstract

Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identifying forbidden actions alone is insufficient to ensure agent safety. In this paper, we argue that agent safety also depends on identifying required yet unperformed safety-critical actions, which we call obligations. Our preliminary study on a popular benchmark for evaluating safety shows that 56.92% of GLM-5.3 trajectories contain unfulfilled obligations, compared with only 30.00% containing forbidden actions. This finding reveals unfulfilled obligations as a major and previously overlooked source of safety risk. However, to our knowledge, no existing benchmark evaluates whether guard models can identify these obligations. To close this gap, we introduce ObligationBench, the first benchmark for evaluating the capability of obligation identification, comprising 240 expert-validated trajectories covering issue resolution, feature development, and terminal operations. Our evaluation of 14 representative models reveals substantial limitations: the highest recall and exact-match rate are only 48.97% and 10.00%, respectively. To address these limitations, we develop ObligationGuard using 40,000 synthetic training examples. ObligationGuard achieves 57.52% recall and an exact-match rate of 21.67%, surpassing all evaluated models on both metrics. We call on the community to incorporate obligation identification into the design and evaluation of future guard models to improve agent safety.