SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
2026-08-25 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors improved a reinforcement learning method called Single-stream Policy Optimization (SPO), which helps AI learn from feedback without needing multiple attempts of the same task. They found that the original method didn't properly balance certain values used during learning, so they adjusted how these advantages are calculated by focusing on the specific actions taken. By also organizing information based on the AI's actions rather than the order it received data, their improved method, SPO++, learns more efficiently on tested problems. Their experiments show that the key improvement comes from normalizing advantages according to action tokens.
reinforcement learningSingle-stream Policy Optimizationtrajectoryadvantage estimationtokenpolicy optimizationvalue estimatenormalizationALFWorldMath-TIR
Authors
Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
Abstract
Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.