Outcome reinforcement learning mostly improves finishing and reliability
What Outcome RL Buys: Finishing, Reliability, Rarely Reach
Computation and Language
Summary
The paper studies how a type of machine learning called outcome reinforcement learning (RL) helps solve physics and math problems. The authors found that outcome RL mainly helps models finish answers within a limited token budget and be more reliable when they do finish. It rarely helps models solve problems they couldn't solve before. They also found that giving extra problem setup information did not clearly improve results beyond standard RL.
What this means in practice
- •For ai model trainers: Improve machine learning models’ reliability and answer completion within fixed computation limits using outcome reinforcement learning.
- •For automated problem solvers: Set realistic expectations for improvements from outcome reinforcement learning when solving complex physics and math problems.
Authors
Shan Yang
Abstract
Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one. On physics and math problems at fixed token budgets, we compare the pass@1 of 20 released RL models (1.5B to 32B) with that of the thinking models they started from. We ask whether each pass@1 gain is changed behaviour or new knowledge. We divide each change in pass@1 into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures problems RL solves that its base answered wrongly in all eight tries, minus problems the base solves that RL answered wrongly in all eight. We find four things. (1) Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing (82% at 4k and 8k). Given more tokens, the base closes most of the finishing gap itself. (2) Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish. (3) By our definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of our four physics and math sets. We checked 144 physics problems that only the RL model solved. Of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. (4) In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.