Training agents to find and follow the best joint convention
ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration
Multiagent Systems
Summary
When computers or robots work together without prior coordination, they need to figure out a common way to act. This paper shows how to train such agents to guess how their partners prefer to work and adjust their behavior accordingly. By training against partners who sometimes change their style and sometimes stick to a fixed way, the agents learn to test their partner’s abilities and settle on the best joint approach. The result is agents that collaborate more effectively with a variety of partners they didn't train with.
What this means in practice
- •For robotic control teams: Develop robots that quickly align with new partners’ teamwork styles for better joint tasks in dynamic environments.
- •For game ai developers: Create game agents that adapt their play style to diverse teammates for improved cooperative gameplay experiences.
Authors
Abhishek Sriraman, Eleni Vasilaki, Robert Loftin
Abstract
Ad-hoc collaboration often requires agents to identify and adhere to some shared convention within a cooperative task. Existing work on reinforcement learning (RL) for ad-hoc collaboration focuses on training agents that adapt to the conventions established by their partners. These methods fail to consider the possibility that while some partners might follow only a single fixed convention, others may themselves be capable of adapting to multiple conventions. Here we present ConventionPlay, an RL-based approach that teaches agents to discover their partner's optimal convention by training against a learned population of partners that exhibit different degrees of adaptability across conventions. Some of these partners follow a single, fixed convention, while others are able to adapt to a subset of the possible conventions for the task in question. The existence of partners that support a limited subset of conventions forces agents trained against this population to actively probe their partner's capabilities, and steer their partner towards the most effective joint strategy that they are capable of following. Our experimental results demonstrate that agents trained via ConventionPlay achieve superior performance to existing ad-hoc collaboration methods against test populations of partners that are compatible with multiple conventions.