Language models struggle to act on partner knowledge in cooperation games
Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions
Machine Learning
Summary
Working together with new partners is hard because you have to figure out unknown ways of communicating. This paper looks at language AI models playing a simplified card game to see if they understand the partner’s intended messages. While the models can often identify the partner’s intent better than the exact rules, they do not always act according to those intentions. The authors show that giving models direct action suggestions improves cooperation more than just stating the rules. This reveals a gap between what these models know about partners and how well they use that knowledge to cooperate.
What this means in practice
- •For game ai developers: Improve AI teammates in cooperative games by integrating direct action suggestions rather than relying on communicated rules alone.
- •For human-robot interaction designers: Design robot systems that better interpret and act on partner communication by distinguishing knowledge from cooperative behavior.
Authors
Yuhwan Jeong, Jinnyeong Yang, Kuk-Jin Yoon
Abstract
Cooperation with unfamiliar partners requires adapting to communication conventions that are not known in advance. We study this problem in a controlled Hanabi-derived environment with scripted hint generation, LLM-controlled receiving decisions, and frozen model weights. Across eight LLMs, linear probes recover intent conventions substantially more accurately than target conventions, yet receiving choices do not consistently agree with the sender's convention. We compare probe-predicted and ground-truth conventions presented either as general rules or as externally computed action recommendations. Rule statements yield modest and model-dependent changes in cooperation, whereas action translation produces larger gains on average. In a Qwen3-8B case study, matched-state statement reversals reveal much greater sensitivity to action recommendations than to rule statements. Activation transfers from oracle-action and non-oracle hint-restatement donors improve intent accuracy on both action classes, but the tested alternatives do not reliably reproduce these benefits. Together, these results distinguish convention decodability, sensitivity to convention information, and cooperative performance, and highlight limitations in turning available partner information into receiving decisions.