Multi agent reinforcement learning improves with staggered participation training
Cooperating with Future Collaborators: Multi-Agent RL under Staggered Participation
Artificial Intelligence
Summary
Sometimes, in tasks where multiple AI agents work together, some start earlier and provide clues that help the later agents. This paper looks at how to better train agents that join in at different times so they can share and use important information more effectively. The authors created a new method called Staggered Participation Learning (SPL) to teach agents how to anticipate and use these clues. They tested SPL in many simulated environments and found it made teams perform better by about 14% on average.
What this means in practice
- •For robotic system developers: Design collaborative robots that coordinate tasks when some start earlier and provide useful information for later robots to improve overall performance.
- •For autonomous vehicle teams: Enhance coordination among vehicles with different start times by training policies to use shared information from earlier vehicles for safer and more efficient travel.
Authors
Jianglin Qiao, Siyi Hu, Thien Hoang Nguyen, Zehong Cao, Salah Sukkarieh
Abstract
In cooperative Multi-Agent Reinforcement Learning (MARL), agents are often trained under concurrent participation, while in many tasks some agents act earlier and leave task-relevant information that becomes useful to agents participating later. We study this setting as staggered participation (SP), which introduces a cross-time, cross-agent learning dependency because an early action may affect the return through the information it provides and the later policy that uses it. Learning under SP therefore requires both identifying what information is useful for future decisions and learning how later agents should use it. We propose Staggered Participation Learning (SPL), a training-time augmentation that addresses these two parts with prospective acquisition supervision for earlier agents and outcome-supervised receiver learning for later agents. We evaluate SPL across multiple policy-based MARL backbones, environments, and staggered-participation patterns. Across 60 MPE/RWARE backbone setting comparisons, SPL achieves higher observed mean task completion in every case, with an average difference of 14.1%. The gains also extend to eight-agent teams and a physics-based UAV-UGV environment in Isaac Lab, providing evidence across algorithmic, temporal, and embodied settings.