Training dynamics reveal how AI and human data differ in learnability
Synthetic Data Characterization via Training Dynamics
Computation and LanguageMachine Learning
Summary
It can be hard to understand how good computer-generated data is for teaching AI compared to human-written data. The authors studied how easy or hard it is for machines to learn from different kinds of data, including data made by large AI models and by people. They looked at training patterns to see what makes some data easier to learn and tested if selecting data based on these patterns helps AI learn better. Their work helps explain what synthetic data can and cannot do in machine learning tasks.
What this means in practice
- •For machine learning engineers: Improve training datasets by selecting easier-to-learn samples for more efficient model development.
- •For data quality teams: Assess and compare the value of synthetic versus human-written data for classification and labeling tasks more accurately.
Authors
Irene Lago, Ana Ezquerro, David Vilares
Abstract
Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.