Large language models improve task focus by adjusting learning pressure
Controlling Sharpening in LLM Post-Training
Machine Learning
Summary
Training large language models more to improve accuracy can sometimes make them worse at handling a variety of tasks, a problem called sharpening. The authors show that certain ways of measuring success, called pass@k and max@k, naturally push the model to focus more on harder tasks and avoid this problem. They provide a new way to use existing training methods with these success measures without needing complex new algorithms. Their method, called KAPPO, efficiently controls sharpening and achieves good performance on tasks like math problems and code writing while saving training time.
What this means in practice
- •For machine learning engineers: Enable controlling task focus in language model training to improve performance on difficult tasks without losing general behavior.
- •For software developers: Apply efficient training techniques that use single completions yet achieve multi-completion standards in mathematical reasoning and code generation models.
- •For conversation ai product teams: Integrate training approaches that balance accuracy and behavioral diversity for better user experiences in dialogue systems.$Commercial implications: Improved training methods can be sold as part of AI model fine-tuning services for conversational products seeking higher reliability on challenging queries.
Authors
Simon Matrenok, Nicolas Menet, Nicolas Flammarion, Andreas Krause
Abstract
LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as $\mathrm{pass@k}$ and $\mathrm{max@k}$ are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for $\mathrm{@k}$ optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from $\mathrm{@}1$ to $\mathrm{@k}$. We derive the optimal policies for KL-regularized $\mathrm{pass@k}$ and $\mathrm{max@k}$ objectives in closed form. Our explicit solutions reveal that $\mathrm{@k}$ optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard $\mathrm{@}1$ objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why $\mathrm{@k}$ objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce $\mathrm{@k}$ Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing $\mathrm{@}1$ methods to $\mathrm{@k}$ via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that $\mathrm{@k}$ pressure can be fully realized with single completion training, matching the performance of multi-completion $\mathrm{@k}$ optimizers at a fraction of the cost.