Data strategies shape capabilities of large language models
Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle
Computation and Language
Summary
Training a large language model depends a lot on how the data is chosen and managed. The authors explain that good training data is not just about what the text says but also how useful that data is at different stages of training. They review many techniques for picking and scheduling data during training and show that methods adapting to the model’s current state can be more effective but are also more complex and costly. They also offer a clear way to categorize these methods and provide a resource listing relevant papers and code.
What this means in practice
- •For machine learning engineers: Optimize training workflows by selecting and adapting data policies that balance model quality and computational cost.
- •For ai platform operators: Implement data management systems that dynamically allocate training data resources to improve large model training efficiency.
A survey. It maps existing work.
Authors
Yuanjian Xu, Jianing Hao, Tianze Sun, Yongyao Duan, Sam Chen, Jinyi Han, Wanbo Zhang, Yuxuan Sun, Guang Zhang, Zhong Li
Abstract
Data strategy plays a central role in training large language models (LLMs). It governs the selection, weighting, and scheduling of training data, thereby determining the capabilities a model can acquire from a fixed data budget. We treat high-quality data as having high training-dependent utility, which depends on the target objective, the model, and its training state rather than on the text alone. Research in this area now spans pretraining and post-training, but related ideas use inconsistent terminology and evaluation protocols. This work reviews data strategies across the training lifecycle and formulates them as policies that allocate a training budget over candidate examples. We organise methods by their decision granularity, model dependence, and temporal adaptivity, then compare their computational requirements and empirical support. Fine-grained or state-dependent policies offer more control but incur additional scoring, search, or rollout costs. Evidence for transfer across model scales and comparable end-to-end cost remains limited. A continuously updated list of papers, along with their public implementations, is available at https://github.com/xuyj233/data_strategy_survey.