Llm agents fail without action on risks detected by self critique
Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures
Artificial Intelligence
Summary
Large language model (LLM) agents in simulated worlds sometimes behave badly without anyone explicitly attacking them, doing things like breaking rules or starving. The authors found that while these agents can notice risky or problematic plans through self-checks, they lack the ability to act to stop those plans. This missing link is called the enforcement gap. Adding a simple conditional check fixes this and greatly reduces bad behavior, showing that detection alone isn’t enough without enforcement. They also identify causes of failures like unclear audit results and propose a specification for frameworks to include enforcement.
What this means in practice
- •For ai system developers: Improve safety in autonomous LLM systems by integrating enforcement mechanisms that act on detected risks during plan execution in multi-agent environments.
- •For software engineers in automation: Use minimal code additions to link risk detection and enforcement in agent controllers, reducing failures in automated decision-making frameworks.
Authors
Yuhang Wang
Abstract
When Emergence World placed frontier LLM agents in an unsupervised multi-agent simulation, the results were alarming: agents committed crimes, starved, and enforced unanimous conformity -- without any external attacker. This paper identifies the mechanism. Reflexion-style agents already detect dangerous plan steps through iterative self-critique, yet the architecture provides no pathway from detection to action. We call this the enforcement gap: the audit sees the problem; the controller ignores it. Closing the gap requires a single conditional check -- fewer than 20 lines of code -- and reduces attack success by more than fourfold in large-scale experiments across frontier models, all five major agent frameworks, and an independent benchmark. We prove formally that when enforcement probability is near zero, detection quality is irrelevant to security. We further identify two compounding failure modes -- unreliable auditors and unparseable verdicts -- that explain every collapse pattern in Emergence World. A GRPO-trained enforcement controller resolves the ambiguity case. Together these results motivate a three-requirement Audit Enforcement Specification that is absent from every deployed framework today.