AutoData improves training data selection for large language models
AutoData: Agentic Search for Pre-training Data Selection
Artificial IntelligenceComputation and Language
Summary
Choosing the right data to train language models is important but usually done by hand. The authors created AutoData, an automated system that tries different ways to pick training data by using small tests to see what works best. AutoData found better data selection methods than people had designed, and these methods also worked well when used in bigger models later. This shows that not just models and code, but data choices can be improved automatically.
What this means in practice
- •For machine learning engineers: Improve language model training by automatically discovering better data selection algorithms using small proxy models.
- •For software development teams: Automate tuning of data preparation processes to enhance AI model quality without manual data curation effort.
Authors
Yan Meng, Dhruv Srikanth, Bingchen Zhao, Zhengyao Jiang, Yuxiang Wu
Abstract
LLM agents have recently shown promise in automating machine learning engineering by editing model and training code under execution feedback. Data, however, remains largely outside this agentic optimisation loop. We frame pre-training data selection as heuristic engineering over per-document features, i.e., lexical statistics, categorical labels, and perplexity. We introduce AutoData, an agent that searches directly over executable selection algorithms. Unlike prior data mixture methods that optimise weights over a fixed set of domains, AutoData searches a richer program space of scoring, stratification, and stochastic selection rules, discovering feature interactions automatically by iteratively refining algorithms with validation feedback from a proxy model. Within an overnight search, AutoData discovers a selection algorithm that outperforms existing human-designed curation pipelines. Despite being searched only on this small proxy, the discovered recipe transfers to larger scales and improves the downstream metric CORE. These results suggest that data engineering can be treated as an agentic machine learning problem, extending autonomous research from model and training-code optimization to the data.