PopGen generates diverse agents for better multi-agent teamwork
PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination
Artificial IntelligenceMultiagent Systems
Summary
When computers learn to work together, training them with partners who behave very differently helps them cooperate better later on. Existing methods measure diversity by looking at actions, which can mistake similar strategies for different ones. The authors invented PopGen, which uses large language models (LLMs) to describe and create truly different agent behaviors in words, then turns those descriptions into working computer programs. Agents trained with PopGen’s diverse populations cooperate better with new partners they haven't seen before.
What this means in practice
- •For multi-agent system developers: Train AI agents that can cooperate effectively with new teammates by generating a wide variety of partner strategies.
- •For game ai programmers: Create diverse NPC behaviors automatically to improve team-based gameplay with unseen player strategies.
Authors
Paul Teiletche, Kanghyun Ryu, Kyungjoon Park, Dhruva Tirumala, Markus Wulfmeier, Negar Mehr
Abstract
In multi-agent reinforcement learning, exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Instead, our key insight is to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce PopGen, a framework that synthesizes semantically diverse agents from scratch using LLMs. PopGen first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that cooperators trained against PopGen populations achieve robust zero-shot cooperation with unseen partners, outperforming population-play baselines across four collaborative environments. We further show that semantic diversity scales with PopGen population size, improving the cooperator's worst-case performance, and we identify design choices that preserve population quality while maximizing diversity. Our code is available at https://github.com/labicon/PopGen.