Language models learn grammar from indirect examples in training data
Where's the Indirect Evidence? Investigating Grammatical Generalization with Influence Functions
Computation and LanguageMachine Learning
Summary
Language models can understand grammar rules even when those rules aren’t shown directly in their training examples. The authors used a technique called influence functions to find which training sentences affected the model’s grammar predictions the most. They discovered that many important sentences didn’t contain direct examples of the grammar but still helped the model learn indirectly. This finding helps explain how language models grasp complex language patterns from indirect clues.
What this means in practice
- •For natural language engineers: Improve training data selection by identifying indirect sentences that improve grammatical accuracy of language models.
- •For language model trainers: Optimize retraining processes by focusing on sentences with high indirect influence on grammar learning across multiple linguistic phenomena.
Authors
William Dai, Zhenghao Herbert Zhou, Maya Viswanathan, Robert Frank
Abstract
Language models generalize beyond their training corpus to learn linguistic phenomena (Patil et al., 2024; Misra and Mahowald, 2024), showing that there exists indirect evidence driving learning. Prior work has inferred that indirect evidence exists without identifying its character or considered specific hypotheses about what it might be. We explore an alternative bottom-up method for identifying indirect evidence. We use influence functions (Koh and Liang, 2017) to measure how individual training sentences affect model predictions in 15 grammatical phenomena from BLiMP (Warstadt et al., 2020). We filter out a phenomenon's direct evidence using hand-constructed rules and take the 60,000 most influential remaining sentences as indirect evidence. This indirect evidence substantially affects learning: retraining without it lowers held-out accuracy more than removing a matched random group for every phenomenon across five models (by a median of 12 points). We characterize the indirect evidence by its similarity to our direct evidence filters: it (partially) meets the filters' structural conditions at a median of 4.7 times the rate of a matched random group. Finally, clustering phenomena by indirect-evidence overlap recovers three of the five BLiMP categories that include multiple phenomena, suggesting that this method can reveal linguistic relationships.