Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
2026-08-13 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a new way to measure how much each piece of training data affects a language model during its learning process without needing to test the model on specific tasks. They do this by seeing how much each data example helps the model's parameters get closer to their final trained state. Using this method on several models, they found that early in training, texts related to literature influence the model more, while later in training, science and math (STEM) data become more important. This shows how different types of data impact learning at different times.
language model pretrainingtraining data influencegradient updatemodel parametersPythia modelsPolyPythiatraining trajectorydata alignmentdownstream tasks
Authors
Yuto Nishida, Hirokazu Kiyomaru, Yusuke Oda, Takashi Kodama, Chaoran Liu, Daisuke Kawahara, Yusuke Miyao, Max Müller-Eberstein, Masaru Isonuma
Abstract
Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.