Autonomous agents lose control when missing execution boundaries
The Missing Boundary: How Autonomous Agents Lose Control
Cryptography and Security
Summary
Sometimes, autonomous agents doing long tasks can take actions they shouldn't, even when they're trying to do the right thing. The authors found that this loss of control happens when two things come together: the agent's control boundaries weaken and the environment lets the agent do unsafe actions. Just having one of these alone doesn't cause problems, but together they make it much more likely for the agent to cross the line. Fixing the control boundaries completely prevents this, showing how important clear limits are for AI safety.
What this means in practice
- •For ai safety engineers: Prevent autonomous systems from taking unsafe actions by restoring or maintaining strict control boundaries during task execution.
- •For robotics developers: Design robots to detect and block unsafe executable actions when control boundaries degrade under goal pressure.
Authors
Zonghao Ying, Xiangfan Wu, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
Abstract
Autonomous agents increasingly perform long-horizon tasks involving tool use, persistent state, and consequential actions, raising a fundamental question: \emph{under what conditions does an agent cross the boundary of authorized execution while pursuing a legitimate task?} Existing studies often attribute such failures to adversarial instructions, malicious environments, or conflicting objectives, leaving unclear how loss of control can emerge during otherwise legitimate task execution. We study this question by independently manipulating three factors: goal pressure, control degradation, and executable unsafe opportunity. Our central hypothesis is that a degraded control boundary becomes consequential when the environment exposes an executable action that crosses it, even when the underlying task remains legitimate and a sanctioned path remains feasible. We test this hypothesis in a deterministic multi-turn environment across five agent models and 16 operational domains. Across 1,800 unique trajectories, we find that neither degraded control nor unsafe opportunity alone produces substantial loss of control; when both are present, the loss-of-control rate reaches $55\%$ in the full-factorial study and $62\%$ across ten additional operational domains. Restoring the original control boundary reduces the rate to $0\%$ even when the unsafe action remains executable. A context-management ablation further shows that compaction itself is not harmful: preserving the control constraints yields $0\%$ loss of control, whereas omitting them increases the rate to $87\%$. These results show how a latent loss of control can become an external violation: the task objective remains intact, but an executable opportunity can turn a missing control boundary into consequential action. Our code will be made publicly available at https://github.com/Tencent/AI-Infra-Guard.