Multi-Agent AI Safety as an Institutional Design Problem
2026-08-10 • Machine Learning
Machine LearningArtificial IntelligenceMultiagent Systems
AI summaryⓘ
The authors study how rules and authority structures in AI systems affect safety during task delegation and resource use. They run thousands of tests with different rule setups and find that both the rules and how authority is tracked influence whether violations happen. Some methods block unsafe actions effectively, but allowing blocked tasks to continue safely is also important. Their work shows that understanding AI institutions requires looking beyond just the formal rules to include the system's state and fallback options.
AI agentstask delegationalgorithmic institutionsmulti-agent systemsauthority stateworkflowrule enforcementprovenanceresource allocationsafety mechanisms
Authors
Abdullah X
Abstract
AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.