Scaling reinforcement learning policies from stored training saves compute
Does Scaling Reinforcement Learning Really Require More Training?
Machine LearningArtificial Intelligence
Summary
Training smarter computer programs with reinforcement learning usually means using more computing power either during training or when the program is running. This paper shows it is possible to get better performing programs from the training history you already have, without extra training or more compute when running them. The authors introduce a method called SURGE that mixes two saved versions of a program to create a new, better one. This method works well on math reasoning and coding tests, improving results beyond what was seen during training.
What this means in practice
- •For machine learning engineers: Improve deployed reinforcement learning models by combining checkpoints from prior runs to achieve better accuracy without extra training or inference costs.
- •For ai service operators: Deploy enhanced RL-based reasoning and coding assistants that deliver stronger results using existing trained models, reducing compute costs in production environments.
Authors
Bangji Yang, Jiajun Fan, Hongba Ma, Ruihan Guo, Ge Liu
Abstract
Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yield policies stronger than the checkpoints visited by its optimizer. We call this policy-space scaling: expanding the deployable policy set accessible from a fixed RL history, without extending training or increasing per-query inference computation. We instantiate it with SURGE (Scaling Up RL Gradient-free via Eigenspace fusion). SURGE combines two checkpoints from the same RL run: a high-accuracy anchor and a competitive donor that generates shorter responses. It expresses both checkpoints as changes from their shared initialization, then spectrally decomposes the anchor's update to retain its dominant component and incorporate the donor's complementary component. With a fixed target for how much of the anchor update to retain, SURGE determines the block size from the weights without testing candidate policies. We evaluate two 1.5B mathematical-reasoning histories, DeepSeek and Nemotron, and one 7B coding history, OLMo. SURGE improves benchmark-average accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. It reaches 54.17% on DeepSeek AIME24 against a measured native maximum of 50.83%, and 83.7% on OLMo HumanEval+ against 82.8%. These gains exceed the observed training curves. Geometric controls support the importance of RL-update structure beyond weight displacement or token reduction alone. Each constructed model runs as a single policy. Our findings identify stored RL history as a reusable scaling resource: the capability available from a training run need not end at its best checkpoint.