Hierarchical multi-agent system improves task solving with joint training
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
Computation and Language
Summary
Solving tasks where one agent leads and others follow can be tricky because the helpers only get indirect feedback on their work. The authors propose a system called MultiAgent-OPD that trains all parts together by combining learned feedback and trial outcomes. They first train a 'Teacher' agent to evaluate correctness, then use it to help guide the helpers. Their approach improves accuracy on multiple tasks over previous methods and allows the team to answer more questions within fewer tries.
What this means in practice
- •For software engineering teams: Develop multi-agent AI systems that coordinate task assignment and execution more effectively with end-to-end training and feedback.
- •For automated customer support teams: Enhance chatbot systems by using hierarchical agents where a lead module assigns subtasks to specialized modules for better answer quality.
Authors
Yuhan Sun, Jiaxin Zhao, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weixun Wang, Weihang Pan, Zhongan Bi
Abstract
In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.