Reinforcement learning improves training stability with adaptive sample weighting
Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling
Machine LearningArtificial Intelligence
Summary
Training AI language models can be tricky because new updates often rely on old data, which can make learning unstable. The authors propose a method called TETPO that adjusts how much importance is given to each piece of data, depending on how reliable it is. This adjustment helps balance the learning process and avoids letting a few samples dominate, which reduces instability. Their approach can find the right level of adjustment automatically and performed better than existing methods in tests involving complex reasoning.
What this means in practice
- •For machine learning engineers: Stabilize off-policy training for large language models to reduce update variance and improve learning consistency.
- •For ai infrastructure teams: Implement adaptive importance weighting to efficiently use stale rollout data without compromising model update stability.
Authors
Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu, Aida Rahmattalabi, Jincheng Cao, Jelena Markovic-Voronov, Alborz Geramifard
Abstract
Asynchronous reinforcement-learning post-training of large language models is computationally efficient but inherently off-policy: the learner updates on rollouts generated by a stale policy, while correcting this mismatch requires importance weights that can be extremely heavy-tailed. A small number of trajectories may then dominate an update, collapsing the effective sample size (ESS) and increasing the variance and instability of the resulting gradient estimates. Existing stabilizers apply fixed interventions regardless of the realized severity of weight concentration, either discarding selected samples or reweighting them without adapting the correction strength. We introduce \emph{Target--ESS Tempered Policy Optimization} (TETPO), which replaces each importance weight $ρ$ by $ρ^α$ for a data-adaptive exponent $α\in[0,1]$: $α=1$ preserves the original importance weights, whereas $α=0$ assigns uniform coefficients. For each prompt group, TETPO selects the largest $α$ whose tempered weights satisfy a target ESS fraction $κ$, applying little tempering to healthy groups and stronger tempering when weight degeneracy is severe. At the population level, the tempered coefficients are exact importance weights for a geometric bridge between the sampler and learner policies. The exponent is efficiently found by one-dimensional bisection, while the resulting weights satisfy deterministic bounds on normalized-weight dispersion and weighted-gradient norm that are independent of policy drift, together with a fixed-$α$ bound on bridge-target discrepancy. Power tempering also introduces no explicit zero-weight mask. On mathematical-reasoning benchmarks under high staleness, TETPO outperforms strong on- and off-policy baselines.