Graph based continual learning improves cooperation in changing multi agent tasks
Continual Graph Multi-Agent Reinforcement Learning
Machine Learning
Summary
When multiple agents have to work together over many different tasks that change, they often forget how to do old tasks when learning new ones. The authors show that using graphs to represent the unique structure of each task helps capture important details. They created a new set of tasks and a method that keeps past knowledge safe by freezing part of the model that understands task graphs. Experiments show their approach helps agents learn new tasks better without forgetting old ones.
What this means in practice
- •For network operators: Manage control policies that adapt across different network topologies, improving robustness over time.
- •For robotics developers: Create multi-robot systems that learn to perform new formations while retaining old configurations efficiently.
Authors
Tommaso Marzi, Ahmed Hendawy, Jan Peters, Carlo D'Eramo, Andrea Cini, Cesare Alippi
Abstract
In Continual Multi-Agent Reinforcement Learning (CMARL), agents learn cooperative policies across sequences of tasks, aiming to adapt effectively to new tasks while preserving the ability to solve previously encountered ones. In many applications, tasks differ in their underlying structure, which can represent, for example, distinct operational conditions or target configurations (e.g., different network topologies in power grids or arrangements in formation control). Existing CMARL methods lack dedicated mechanisms to leverage this structural information when learning new tasks, failing to promote transfer and mitigate forgetting. To fill this gap, we propose Continual Graph Multi-Agent Reinforcement Learning (CGMARL), a novel framework for CMARL problems in which task sequences are mapped into a series of attributed graphs, each modeling a task-specific structure. In CGMARL, each graph determines the environment dynamics (next states and/or rewards) and the number of agents for the corresponding task. Then, we present Graph-based Formation (GRAFO), the first CGMARL benchmark, and show how forgetting arises in this setting. Finally, to address this limitation, we propose Frozen Graph Encoder (FROG), a method that relies on a frozen graph backbone to preserve past structural information in graph-based CMARL policies. Experiments on GRAFO show that pairing FROG with existing CL methods substantially improves performance on multiple CGMARL scenarios.