Summary
Language models are tested by comparing pairs of sentences where one is grammatically correct and the other is not, but usually these tests don't consider how often the words appear in real life. The authors created a new test called FreqBLiMP that checks if language models still recognize correct grammar when rare words are used. They found that as words become less frequent, models are less confident, but accuracy only drops a little overall. However, the models struggle more with grammar tasks that need specific word knowledge while doing well on simpler grammar rules. This study highlights how word rarity can affect how well models understand language.
Language modelsMinimal pairsLexical frequencyGrammatical acceptabilityZipf frequencyMorphosyntaxLemmaSentence likelihoodContrastive evaluation
Abstract
Minimal-pair benchmarks such as BLiMP evaluate linguistic knowledge by testing whether language models (LMs) prefer acceptable sentences over minimally different unacceptable ones. However, these benchmarks largely ignore lexical frequency variation, despite lexical frequency being a pervasive and highly skewed property of natural language use. Consequently, existing evaluations do not test whether grammatical preferences remain stable when contrasts involve rare lexical items. We introduce FreqBLiMP, a frequency-controlled extension of BLiMP that regenerates all 67 paradigms under explicit Zipf-frequency regimes while preserving each minimal-pair's grammatical contrast. Evaluating multiple open-weight LLM families across scales, we find that decreasing lexical frequency produces a consistent, monotonic decrease in sentence likelihood, but only a modest reduction in overall contrastive acceptability accuracy. However, this aggregate stability masks substantial variation across linguistic phenomena, with LLMs remaining robust on overt morphosyntactic generalization while degrading on phenomena that require lemma-specific information.