Dynamic Important Example Mining for Reinforcement Finetuning
Artificial Intelligence
Summary
The authors look at a way to improve how large AI models learn to reason by changing how training examples are chosen during reinforcement fine-tuning. They notice that previous methods pick training samples in a fixed way, which doesn’t account for how learning changes over time. To fix this, they propose a new method called DIEM that dynamically picks the most helpful examples at each training step based on how much they improve the model. Their approach adjusts example weights carefully to keep training stable and shows better results on reasoning tasks compared to older methods.
Authors
Haoru Tan, Sitong Wu, Yanfeng Chen, Shizhen Zhao, Yang-Tian Sun, Tianjia Liu, Chirui Chang, Shaofeng Zhang, Samm Sun, Xiuzhe Wu, Ruobing Xie, Xiaojuan Qi
Abstract
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.