Summary
Many AI systems struggle to figure out hidden rules in changing social settings where they interact with others multiple times. The authors created OpenSocialJax, a test system where AI agents face millions of different hidden environments that affect how they can work together or compete. They tested several large AI language models to see if they could learn these hidden rules and cooperate with others. While some agents discovered important rules and improved their play, cooperation was often unstable, with resources running out or cooperation burdens falling unevenly on some agents. This work shows how hard it is for AI to adapt together in complicated, changing social worlds.
What this means in practice
- •For multi-agent system developers: Evaluate how AI agents discover hidden rules and adapt cooperation in complex social environments using OpenSocialJax scenarios.
- •For game designers: Design more dynamic game environments that test player adaptability and cooperation over repeated interactions with hidden mechanics.
Authors
Zihao Guo, Meng Fang, Joel Z. Leibo, Yali Du
Abstract
Sequential social dilemma benchmarks capture diverse social interactions, but rarely test whether agents can infer hidden environmental rules through repeated interaction while adapting to other agents. We introduce \textbf{OpenSocialJax}, which extends Clean Up and Commons Harvest across an open-ended distribution of latent environment configurations, systematically varying spatial layouts and hidden rules governing tool crafting, payoffs, and resource dynamics. Each episode samples one configuration from tens of millions of possibilities and holds it fixed across five trials. We evaluate eight large language models, including three closed source models and five open source models. Our evaluation tests whether models can discover hidden rules and adapt to other agents in homogeneous and mixed teams. We find that rule discovery often improves performance but does not reliably sustain cooperation, as the cleaning burden becomes concentrated among fewer agents in OpenCleanup and every team eventually exhausts the shared resource in OpenHarvest. In our mixed arena, one cooperative agent was able to sustain the public good in OpenCleanup at a cost to its own return, but could not prevent resource depletion in OpenHarvest. Together, these results highlight the difficulty of adapting jointly to unfamiliar environmental dynamics and the changing behavior of other agents.