Measuring hidden teamwork signals in AI and human card play
The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation
Artificial IntelligenceHuman-Computer Interaction
Summary
This paper looks at how humans often cooperate by sharing unspoken signals—called conventions—that go beyond the obvious words or hints used in a game, something AI players usually miss. The authors introduce the idea of a "convention gap," which measures how much human teamwork relies on these hidden signals by comparing expected failures based on literal clues to actual failures observed. Using the card game Hanabi, they found large gaps in human pairs and mixed human-AI games but not among AI players alone. This shows that AI partners might need to better understand and align with human conventions to work well together.
What this means in practice
- •For game ai developers: Design AI agents that better detect and use human implicit communication patterns to improve teamwork in multi-player games.
- •For human-robot interaction designers: Create robots or AI systems that adapt to human implicit conventions, increasing collaboration success in shared tasks.
Authors
Makoto Fukushima, Hua-Dong Xiong, Ehsan Moradi Pari
Abstract
Cooperative AI agents are evaluated against other AIs, yet human cooperation relies on implicit conventions---shared protocols for reading meaning beyond the literal message---which AI-AI benchmarks may not capture. We propose the \emph{convention gap}, the difference between the failure probability predicted from the literal content of communication and the observed failure rate, as a metric of implicit communication. In the card game Hanabi, the finite deck and deterministic hint constraints make this posterior exactly computable. We replayed about 101,000 play actions from three public datasets of human-human (hanab.live), AI-AI (HOAD), and human-AI (HanabiData) games. The gap was +26.2 percentage points (pp) in human pairs, $-$0.7~pp in AI pairs, and +16.4~pp in human-AI pairs, and was concentrated on plays of cards that had received no hints (+46~pp in human pairs). Within human-AI play, the literal information available to humans was similar across the three AI partners (mean predicted failure 38--41\%), but human failure rates ranged from 14.4\% to 34.4\% and the gap from +24.1 to +6.2~pp; the partner eliciting the largest gap produced the fewest human failures. Game score carried different information: it depended on each corpus's roster composition, whereas the gap separated human from AI play at the agent level. As a known-answer check, Off-Belief Learning agents, whose convention content is controlled by construction, gave a gap of +1.6~pp at the convention-free level, rising monotonically to +21.7~pp. These results suggest that convention compatibility, rather than AI-AI performance, may predict an AI's effectiveness with human partners.