Multi agent language models learn better by updating shared actions together
SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
Artificial Intelligence
Summary
When several language models work together, they often make decisions that affect the same situation at the same time. Existing methods update each model’s choices separately, which can miss how these choices combine to change things. The authors proposed SRPO, a new way to treat a group of decisions as one action to improve learning and cooperation. Their experiments showed this approach works well for tasks like math problems and search tasks, improving stability and performance across different setups. This could help multi-agent AI systems coordinate more smoothly and effectively.
multi agent systemslarge language modelsreinforcement learningpolicy optimizationstate transitionjoint actionslog ratiosadvantage functionworkflow routing
Authors
Shengtian Yang, Ziyu Xiong, Yu Li, Yewen Li, Qingpeng Cai, Lei Feng
Abstract
Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.