Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
2026-07-06 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors studied how to reuse high-quality training data when teaching large language models. They found that instead of just training once, repeating data multiple times can help the model learn better, but too much repetition might cause problems. By watching how the model remembers information over time, they created a method to decide the best times and amounts to reuse data during training. Their early tests show it's beneficial to repeat data more than what people commonly do. This work lays the groundwork for smarter training schedules that make the best use of limited good data.
large language modelstraining paradigmdata reuseoverfittingmemorization windowloss retentionmodel performancesample efficiencytraining epochsdata replay scheduling
Authors
Jingwei Zuo, Cong Zeng, Ilyas Chahed, Maksim Velikanov, Dhia Eddine Rhaiem, Pasquale Balsebre, Abhay Kumar, Younes Belkada, Hakim Hacid
Abstract
The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's "Memorization Window" signals derived from loss retention dynamics and downstream evaluation scores, we propose "Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.