Centralized critics improve teamwork for many-agent learning systems
Centralized Critics for N-Agent Ad Hoc Teamwork
Artificial IntelligenceMachine Learning
Summary
When multiple agents work together in a team but have to coordinate with unknown partners, it’s hard to teach them how to cooperate well. The authors study two ways to help learning agents judge their actions using information from their own history alone or also from the histories of their unknown teammates. They prove both methods are unbiased but show that using joint histories can reduce errors during learning. Their experiments find that considering teammates’ histories often leads to better team coordination and faster learning.
What this means in practice
- •For robotics teams: Enhance coordination among robots working with unknown partners by incorporating teammates' action histories into centralized critics during training.
- •For autonomous vehicle fleets: Improve learning efficiency for fleets adapting to new vehicles by using joint historical data from all agents to better estimate value during training.
Authors
Andrew McAlinden, Caroline Wang, Steven Shi, Arrasy Rahman, Elad Liebman, Peter Stone
Abstract
Centralized training with decentralized execution (CTDE) has become the dominant paradigm in multi-agent reinforcement learning (MARL), motivating extensive study of what privileged information should be made available to the critic during training. While prior work has examined the role of global state in fully cooperative settings, the design of centralized critics for N-agent ad hoc teamwork - in which dynamically varying set of learning agents must coordinate with an arbitrary set of uncontrolled teammates drawn from an unknown distribution - remains poorly understood. We study this question by comparing two natural classes of advantage estimators: controlled history estimators, which condition only on the observation and action histories of the learner's own agents, and joint history estimators, which additionally condition on the histories of the uncontrolled teammates. We prove that both estimators are unbiased and characterize the conditions under which the joint history estimate attains strictly lower variance than its controlled counterpart. Empirically, joint history critics match or outperform controlled history critics across a range of NAHT tasks, yielding improved sample efficiency and final returns. Together, these results provide both a theoretical justification and an empirical recommendation for incorporating uncontrolled agent histories into centralized critics for N-agent ad hoc teamwork.