Combining judges improves detection of harm in AI agent environments
Widening Harm Evaluation Coverage with a Judge that Reads the World
Artificial Intelligence
Summary
AI agents can cause harm that shows up either in what they say and do or in how they change their surroundings. Existing safety checks look at only one place, either the agent's actions or the final state of the world. The authors created a system where a judge uses both the agent’s actions and the environment’s state to find harmful outcomes. They found combining these two views catches more harmful cases than looking at either alone. Yet, using one judge that sees both does not work as well as combining two specialized judges.
What this means in practice
- •For ai safety teams: Detect more safety hazards by combining separate judgments on agent actions and environment state changes in evaluations.
- •For game developers: Improve automated identification of harmful player or AI behaviors using combined analysis of game history and current world state.
Authors
Prabhpreet Singh Setia, Bhavesh Neekhra, Debayan Gupta
Abstract
An agent acting in a stateful environment can leave evidence of harm in two places: the record of what it did and said, or the world it changed. Safety evaluations for discovering harm examine one or the other in limited ways, using either an LLM judge over the trajectory or deterministic checks over the final state. Although LLM-based verifiers that inspect the environment exist, they have been used to verify task completion, not to detect and confirm harm in a safety setting. We address this gap with an LLM that has read-only access to the environment state and is given a harm taxonomy. We evaluate this approach in three environments: two from DecodingTrust-Arena and one from OSWorld. We hold the judge model and harm taxonomy fixed, varying only what the judge observes: the trajectory, read-only tools for inspecting the environment, or both channels together. In the two DecodingTrust-Arena environments, combining the trajectory and state-only judges recalls 94 percent of confirmed harms, compared with 53 percent for the trajectory judge alone. On OS-Harm, the union recalls 77 percent, compared with 67 percent. A single judge given both channels does not perform better and produces more false negatives than the state-only judge. We also record disagreements and find that the two channels miss different episodes. These results suggest that combining verdicts from two channel-specific judges is a robust and flexible approach to detecting and verifying harm in agentic settings.