Reinforcement learning failures pinpoint small text clues to fix base models
Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining
Computation and LanguageArtificial Intelligence
Summary
Language models trained with reinforcement learning sometimes fail because they miss key reasoning steps. The authors found that specific short phrases in the model’s failed answers strongly predict these failures. By finding similar text in the original training material related to these failure phrases and continuing to train the model on it, they improved the models’ abilities. This approach uses much less training data than usual and works across different models and types of questions.
What this means in practice
- •For machine learning engineers: Improve language model reasoning by identifying and retraining on small sets of failure-related text snippets before reinforcement learning.
- •For ai product developers: Enhance AI reasoning features in math and science question answering by targeted continued pretraining of base models.
Authors
Xinyi Wang, Danqi Chen
Abstract
Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure $n$-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3--4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by $2.0$ points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by $1.9$ points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.