Multi-agent learning improves using frozen single-agent policies
MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies
Multiagent SystemsRobotics
Summary
Training multiple agents to work together can be tricky because they have to learn to complete a task alone and also coordinate with others at the same time. The authors found a way to reuse existing single-agent skills by transforming what each agent sees into a format their single-agent brain already understands. This means the agents can cooperate better without relearning everything from scratch. Their method uses much fewer new learnable parts and works well even when team sizes change.
What this means in practice
- •For robotics engineers: Enable robot teams to coordinate efficiently by reusing existing single-robot control policies with minimal retraining for multi-robot tasks.
- •For game ai developers: Create multi-agent game characters by adapting single-character behaviors, reducing training time and maintaining strong team coordination.
Authors
Elie Abboud, Oren Gal
Abstract
Multi-agent reinforcement learning (MARL) commonly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordination simultaneously. Yet many multi-agent problems admit a compatible single-agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-agent policy, inducing actions suited to the shared environment without updating the single-agent policy itself. MATES leaves the pretrained policy's internal architecture unchanged and retains the objectives and update procedures of the underlying MARL algorithm. We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5-7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration-based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi-agent behavior can be learned without modifying the policy that encodes individual competence.