Reinforcement learning sharpens answers without improving reasoning skills
RLVR Sharpens the Pencil, Not the Mind
Machine Learning
Summary
Some AI models get better at solving problems when rewarded for correct answers, but it isn’t clear if they actually learn to reason better. The authors watched one model during training and found it already knew the necessary reasoning—it just got better at choosing the right answers. The model’s internal understanding of mistakes did not improve, only its final selections. This means the training helps the model pick its best thinking, not create new or clearer thoughts.
What this means in practice
- •For ai model trainers: Optimize reinforcement learning setups knowing training sharpens output selection without changing internal reasoning.
- •For automated testing engineers: Use internal probes to verify AI systems maintain reasoning coverage even if accuracy improves during tuning.
Authors
Shreeyam Bangera, Manya Gupta
Abstract
Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening---that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.