Multi-agent systems learn clearer roles and teamwork with on-policy distillation
MAS-OPD: On-Policy Distillation for Multi-agent Systems
Computation and Language
Summary
Splitting tasks among multiple specialized agents helps solve complex problems, but getting them to work well together is hard. The authors introduce MAS-OPD, a new way to teach these agents by giving detailed feedback from a teacher system during their learning. This approach helps each agent understand its role better and coordinates teamwork without needing complex, task-specific rewards. Tests on coding and math tasks show that MAS-OPD outperforms other methods and leads to better cooperation among agents.
What this means in practice
- •For software development teams: Use MAS-OPD to improve multi-agent AI systems that automate coding tasks by fostering specialized roles and better collaboration without redesigning rewards.
- •For automated math solving platforms: Integrate MAS-OPD to train multi-agent solvers that more effectively divide and conquer mathematical problems with enhanced coordination.
Authors
Qiyong Zhong, Mao Zheng, Mingyang Song, Houcheng Jiang, Jiajie Su, Huwei Ji, Li Zhang, Junfeng Fang
Abstract
Multi-agent systems (MAS) split a task across specialized roles and are promising on complex tasks, yet a prevailing approach relies on inference-time orchestration alone. General-purpose APIs are costly and hard to customize, while small models with role prompts rarely develop stable role competence or reliable collaboration, so post-training a MAS jointly is central. Most attempts use reinforcement learning, whose team-level reward leaves undetermined which step of which agent brought about the outcome, while local rewards need redesigning per task. On-policy distillation (OPD) gives token-level teacher supervision on trajectories the student samples, a denser signal needing no local reward, yet is underexplored for the interdependent agents of a MAS. Two difficulties arise: building complementary specialization from a judgement of which role a behavior belongs to while preserving the knowledge all roles need, and turning cross-agent collaborative information into supervision OPD can exploit. We present MAS-OPD, where Role-Advantage Specialization defines the role advantage as the difference between the teacher signals under target and non-target role conditions, and Privileged Attribution for Coordination attributes an interaction conflict to its source and supplies it to the teacher alone as privileged information. Extensive experiments on code and mathematics benchmarks show that MAS-OPD attains the highest mean score at both student scales and leads the agents to develop clearer role specialization and more effective collaborative behavior.