Multi-agent models improve coordination with online fine-tuning
MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization
Artificial Intelligence
Summary
Many groups of AI agents learn how to work together by studying past data, but they have trouble when new situations come up. The authors propose a method that lets these agents learn more by practicing together in a live environment after their initial training. This approach helps them better adapt and cooperate when facing new challenges. Their method showed clear improvements over previous ways of training agents both offline and online.
What this means in practice
- •For robotics teams: Enhance robot teams to adapt and coordinate better in changing environments by combining offline training with online interaction-guided fine-tuning.
- •For game ai developers: Create game opponents or allies that improve their cooperation dynamically after initial data training, leading to more challenging and realistic gameplay.
Authors
Guowei Zou, Haonan Chen, Haitao Wang, Beiwen Zhang, Na Yan, Hejun Wu
Abstract
Multi-agent flow policies learn cooperative behavior from fixed offline datasets, but often struggle to complete tasks in situations not covered by the offline data. In these situations, agents must both adapt to changes in the environment and coordinate with one another, yet action patterns learned offline are often insufficient for effective adaptation and coordination. To address this problem, we propose Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), which uses online fine-tuning to improve the cooperative behavior of models pretrained with flow matching through new interactions with the environment. Building on the behavior learned during pretraining, we construct policies with explicit action likelihoods for discrete and continuous action spaces. We then update the pretrained model using shared team advantages to further improve coordination based on team performance. Our method achieves, on average, relative gains of 52.8% over the strongest listed offline baselines across 30 settings and 29.8% over purely online learning across 38 comparisons with matched online budgets and evaluation protocols.