Language models learn category-based reasoning during training
Latent Taxonomies: How Property Induction Develops in Language Model Pre-training
Computation and LanguageMachine Learning
Summary
People tend to guess that things similar to robins, like sparrows, share certain facts, but not things very different, like whales. The authors studied whether large language models behave this way when learning from text alone. They found that models start to group related items and share properties within those groups, showing a kind of reasoning like people do. Larger models didn’t always get better at this, especially if they didn’t get more data. This helps us understand how language models generalize facts based on patterns in their training data.
What this means in practice
- •For data curation teams: Manage training datasets to promote useful generalization by recognizing and structuring latent hierarchical categories in data.
- •For prompt engineering teams: Design prompts that exploit or override a model’s learned category structures to control fact propagation and generalization behavior.
Authors
Jan Sobotka, Ekaterina Goliakova, Marcin Detyniecki, Antoine Bosselut, Xavier Renard
Abstract
When told that robins have sesamoid bones, people are far more prone to extend the property to sparrows than to whales. This behavior, called category-based property induction, is well documented in humans, but whether and how it develops in language models trained only to predict the next token is unclear. To shed light on this, we construct a hierarchical taxonomy of synthetic entities grouped by the properties they share. We then inject facts from this latent taxonomy into the continued pre-training of Pythia and OLMo 2 models from 70M to 7B parameters, and track the development of inductive behaviors. We find that taxonomically aligned induction emerges from next-token prediction alone: a property learned for one entity transfers more strongly within its taxonomic category than across categories. Internally, entities that share a deeper category become more similar in activation space, and clustering recovers the full taxonomic tree from entity representations. Contrary to the usual expectation that scale helps, we find that induction weakens with model size when the data budget is fixed, suggesting that larger models memorize individual facts rather than compress them into shared structure. Induction also strengthens with more pre-training before the taxonomy injection, transfers to novel in-context properties, and can be overridden by a latent taxonomy implied by the prompt. Together, these results show that a model's inductive generalization reflects the latent organization of its training data, with implications for data curation and for predicting how far a learned fact will spread.