Non-language data helps train language models but with limits
Structural priors for data-efficient language learning
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Training language models usually needs lots of text data and big computers. This paper studies if training on other kinds of structured data like music or patterns can help language learning. The authors found that starting training on these non-language data reduces the work needed later and improves some training signals. But, this approach does not always help the models perform better on language tasks and is not as effective as just using more language data. So, other data types can partly replace language examples but don’t fully teach language skills.
What this means in practice
- •For machine learning engineers: Use training on symbolic data like music or structured patterns to reduce time and data needed when initializing language models.
- •For natural language processing teams: Improve initial language model training setups by incorporating non-language data to achieve lower initial loss and faster convergence.
Authors
Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn
Abstract
Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization.