RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
2026-07-06 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a challenge in teaching large language models to complete multi-step tasks using reinforcement learning, where rewards are rare and training can be slow or ineffective. They propose a new method called Reward-Swap Policy Optimization (RSPO) that uses more frequent, detailed rewards to guide learning while still focusing on the true final goals. This approach helps the model explore more varied solutions without getting confused by misleading feedback. Their tests on two task sets show that RSPO improves performance across different learning algorithms. Overall, the authors found a better balance between learning speed and accuracy in long tasks.
reinforcement learninglarge language modelsmulti-turn taskssparse rewardsdense rewardspolicy optimizationreward shapingWebShop benchmarkALFWorld benchmarkPPO algorithm
Authors
Qiang Liu, Taian Guo, Ruizhi Qiao, Xing Sun
Abstract
Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model's final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.