ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
2026-06-29 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how multiple agents can work together to achieve a common goal in reinforcement learning. They found a way to break down the overall task so each agent can update its strategy independently, yet still improve the group’s performance together. Their method, called Agent-Chained Policy Optimization (ACPO), treats the group’s joint decision as a sequence, where each agent considers the others’ actions in order. Experiments show that ACPO performs better than existing methods, especially when more agents are involved.
Multi-Agent Reinforcement LearningCentralized Training Decentralized ExecutionPolicy GradientJoint Policy GradientDecentralized CriticsAgent-Chained Policy OptimizationScore FunctionNash EquilibriumValue DecompositionMulti-Robot Warehouse
Authors
Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara, Scott Sanner, Pascal Poupart, Jongmin Lee, Kee-Eung Kim
Abstract
Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.