Tandem balances critics to improve multi-agent learning in complex settings
TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments
Multiagent Systems
Summary
Training multiple agents to cooperate can be tricky because they learn using information they won’t have when they actually act. This gap gets worse when agents have only partial or noisy views or when the group changes during a task. The authors introduce TANDEM, a method that simultaneously uses both a full global viewpoint and the limited local viewpoints during training. It decides when to trust local views and when to rely on global info, helping agents learn better policies that work well when they act alone. This approach beats existing methods in several challenging scenarios.
What this means in practice
- •For autonomous vehicle teams: Coordinate multiple self-driving cars to act effectively despite limited shared sensor data during operation.
- •For emergency response units: Improve coordination among robots or drones for wildfire suppression where team members have partial and noisy information.
Authors
Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi
Abstract
Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.