Monofacts in training data boost large language models scientific discoveries
Tracing the Scientific Discovery Capacity of LLMs to Their Training Data
Computation and LanguageArtificial Intelligence
Summary
Large language models can help make new scientific discoveries, but it's unclear what in their training helps them do this reliably. The authors found that facts seen exactly once during training, called monofacts, play a key role. By increasing the number of monofacts in training data, the models made more discoveries in areas like materials science and genomics. This effect isn’t just due to having diverse data and works across different scientific domains when combined properly.
What this means in practice
- •For machine learning engineers: Improve scientific discovery by including more monofacts in training datasets to enhance large language model capabilities.
- •For biotech data teams: Boost genomic data analysis with language models trained on data enriched with unique facts for better discovery outcomes.
Authors
Arda Uzunoglu, Mahler Revsine, Michael Schatz, Benjamin Van Durme, Mehrdad Farajtabar, Daniel Khashabi, Eric Nalisnick
Abstract
Large language models (LLMs) show growing promise for scientific discovery, yet basic questions remain about what enables reliable discovery. How can we trace the scientific discovery capacity of LLMs to their training data? We study this question as one of missing probability mass and develop a theoretical account in which monofacts-the facts observed once during training-set the discovery capacity. In materials science and regulatory genomics, we directly manipulate monofact rates in training sets and find that more monofacts consistently yield more discoveries across two open-weight model families, sampling strategies, and reasoning budgets. Further experiments show the relationship cannot be explained by training-set diversity alone and can transfer across scientific domains when paired with in-domain data. Together, our results identify monofact rate in training data as a key factor governing discoveries in language models.