Setwise policy updates improve multi-agent language model coordination
SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
Artificial Intelligence
Summary
Multi-agent large language models often work together to solve complex tasks, but current methods update each agent's actions individually, which can miss how these actions combine to cause results. The authors propose SRPO, a way to treat the combined actions of multiple agents as a single unit, improving how the model learns to coordinate. They tested SRPO on tasks like math reasoning and multi-turn search, showing stable training and better overall performance compared to other methods. This approach helps models handle different teamwork arrangements more effectively.
What this means in practice
- •For ai development teams: Improve training of multi-agent language models that solve complex tasks through coordinated outputs.
- •For search engine developers: Enhance multi-turn search systems by better coordinating large language model agents for improved query responses.
Authors
Shengtian Yang, Ziyu Xiong, Yu Li, Yewen Li, Qingpeng Cai, Lei Feng
Abstract
Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.