LinMix improves language model training by better data domain weighting

LinMix: Data Mixing as Linear Domain Attribution

Machine LearningComputation and Language

Summary

Language models learn from lots of text data that come from different types of sources called domains. How much each domain contributes to training affects the model's performance. The authors show that common ways of mixing these domains rely on poor sampling strategies that make it hard to tell which domains truly help. They introduce LinMix, a method that selects extreme mixes where each domain is either fully included or not, making it easier to measure each domain's impact. Tests with large language models showed LinMix leads to better results in tasks like coding and math.

What this means in practice

  • For machine learning engineers: Optimize the selection of domain data mixes during language model pretraining to improve performance on reasoning, coding, and math tasks.
  • For data scientists in finance: Use vertex sampling to attribute and weigh data sources for training models on diverse financial datasets where understanding domain contributions improves prediction accuracy.

Authors

Hubert Baniecki, Przemyslaw Biecek, Martin Pawelczyk

Abstract

Language model performance depends strongly on the mixture of domains in the pretraining data corpus. Popular data mixing methods select these domain weights by training many small proxy models on random mixes and then fitting a regression model to predict performance from the mix. We recognize this practice as a domain attribution problem, where the regression measures each domain's contribution to the target metric. Through this lens, the standard Dirichlet sampling of proxy data mixes is a poor regression design that requires tuning and cannot support a linear fit for hundreds of domains. Building on classical optimal experimental design, we introduce LinMix, which samples mixes at the vertices of the capped simplex so that each domain is at its cap or absent. This hyperparameter-free vertex sampling decorrelates domain effects, enabling an interpretable linear model whose coefficients are the domain attributions. The optimal mix given by LinMix provably keeps only domains with positive attribution values and requires no search over candidate mixes. We further establish the optimality of vertex sampling, and prove that Dirichlet sampling inflates the attribution error when its mixes concentrate around the prior data mix. Pretraining experiments with 1B-parameter OLMo2 and 4B Qwen3.5 language models show that LinMix outperforms Dirichlet-based data mixing across reasoning, code, and math benchmarks, with relative gains of 46% on MBPP and 14% on GSM8K. At 288 fine-grained domains, only vertex sampling supports a linear fit, and LinMix improves benchmark accuracy by 11% on average over the best nonlinear baseline.