Language model circuits for grammar get simpler during training
The Emergence and Sparsification of a Syntactic Circuit
Computation and Language
Summary
This study looks at how language models learn to agree on subjects and verbs, like matching singular and plural words. The authors found that these models start with many neurons handling this agreement, but over time, the responsibility shifts to fewer, more specific parts of the model. Interestingly, which parts handle the task can change during training, and the final system is surprisingly simple. This helps us understand how language skills form inside AI models.
What this means in practice
- •For ai model developers: Optimize language models by targeting smaller, specialized circuits for grammar tasks to improve efficiency and interpretability.
- •For nlp system engineers: Design grammar-related components based on how subject-verb agreement circuits develop and reorganize during training to enhance model robustness.
Authors
Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald
Abstract
To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.