Outcome-Mass alignment improves multi-turn language model agent learning
Outcome-Mass Alignment: Controlling Length-Dependent Trajectory Influence in Multi-Turn LLM Agent Reinforcement Learning
Computation and Language
Summary
When teaching AI agents that hold long conversations or perform multiple steps, it's tricky to fairly measure how their actions lead to good or bad results. The authors highlight problems that happen when counting successes multiple times or giving unequal importance to different action sequences. They present a technique called Outcome-Mass Alignment (OMA) that fixes these counting issues without changing the basic learning process. OMA led to better success rates in tests with simulated environments where longer failure paths expanded more than successes.
What this means in practice
- •For ai system developers: Improve reinforcement learning outcomes for long interaction tasks with large language model agents by using outcome-statistics correction techniques.
- •For conversational ai teams: Enhance dialogue agent training reliability where multi-turn conversations produce uneven data expansions affecting success measurement.
Authors
Dawei Zhang, Yubin Huangfu, Jessica Fan, Boyu Wei, Likang Wu, Hongke Zhao, Ming He, Zhu Teng
Abstract
To keep long interaction contexts tractable, stepwise group-relative training for multi-turn LLM agents materializes each source trajectory into multiple response rows and valid tokens. This expansion creates an outer-accounting problem distinct from within-trajectory credit assignment: row-based outcome statistics can count one episode outcome multiple times, while token-level reduction can assign unequal aggregate objective mass across trajectories. We term these effects outcome-statistics multiplicity and objective-mass imbalance. We introduce Outcome-Mass Alignment (OMA), a two-component method that makes both accounting choices explicit. Outcome Alignment computes outcome statistics over unique source trajectories, while Mass Alignment controls aggregate objective mass after within-trajectory aggregation of optimizer-native token objectives. Neither redesigns the base optimizer's credit assignment, importance ratios, or clipping. Across matched ALFWorld GRPO comparisons at two model scales, where failed trajectories typically expand much more than successful ones, OMA improves matched-horizon endpoint success by 7.0-9.4 percentage points. Ordinary WebShop exhibits weak or reversed expansion-outcome coupling and correspondingly mixed effects, whereas in a success-oriented WebShop setting, WebShop-SOL, OMA improves success nAUC from 0.399 to 0.511. Component ablations and GiGPO and GSPO comparisons further support component separability and cross-optimizer compatibility. These results show that trajectory expansion always defines an implicit weighting rule, but its practical effect depends on how expansion couples to outcomes and optimizer-native updates.