Language model teams struggle with missing communication and perspective gaps
Sanzaru: Benchmarking Protocol Formation under Epistemic & Communication Asymmetry
Computation and LanguageMachine Learning
Summary
If a group of AI agents work together on a task but each has different communication limits — like one cannot hear, another cannot speak, and a third cannot see — solving puzzles becomes much harder. The authors created a benchmark called SANZARU to test how well teams of three such agents coordinate to defuse bombs using limited shared information. Even strong AI models succeeded less than half the time, often because they tried to communicate in ways their teammates could not understand. This work helps measure how well AI teams cooperate when communication is restricted and different roles have unequal information.
What this means in practice
- •For multi-agent system developers: Design and evaluate communication strategies for AI teams working with unequal information and restrictions on message types.
- •For human-robot interaction teams: Improve cooperation protocols for robots or assistants when team members cannot share the same communication channels or perspectives.
Authors
Yiping Li, Zhiyu An, Wan Du
Abstract
A language model's ability to solve a task individually does not ensure that a team can solve it under unequal access to information and communication. SANZARU examines this gap through cooperative bomb-defusal puzzles adapted from BOMBANANA! (Lefto Studio, 2026). Task information and action authority are divided among three agents: one cannot hear, another cannot speak, and the third cannot see. The benchmark contains 120 puzzle instances across seven families, each targeting a distinct coordination ability. Across 1512 episodes, 7 models achieve an overall success rate of 49%, ranging from 20% to 86% under the evaluated configurations. Teams develop or adopt diverse communication protocols, from simple gesture codes to Morse and encodings of entire manuals. Yet trace analysis attributes about 60% of 771 team losses to communication, and agents attempt to use channels unavailable to their teammates despite explicit role instructions. Together, task outcomes and interaction traces enable evaluation of both collaborative performance and perspective taking throughout cooperation. The benchmark is available at https://github.com/markli404/sanzaru-benchmark.