Language models struggle more with rare words but keep some grammar skills
FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity
Computation and LanguageArtificial Intelligence
Summary
Language models are tested to see if they can tell good sentences from nearly identical bad ones. The authors found that when sentences have rare words, these models give them lower probabilities but still mostly pick the right grammar differences. However, the models do worse with grammar that depends on specific word knowledge and stay strong only on common grammar rules. This new test helps reveal where language models are sturdy and where they become fragile with unusual vocabulary.
What this means in practice
- •For natural language processing teams: Identify where language models fail on rare words to improve their handling of uncommon vocabulary in applications like text generation.
- •For machine translation developers: Use frequency-controlled tests to assess and enhance translation models’ accuracy on sentences containing rare lexical items.
Authors
Tyrone White, Yuki Arase
Abstract
Minimal-pair benchmarks such as BLiMP evaluate linguistic knowledge by testing whether language models (LMs) prefer acceptable sentences over minimally different unacceptable ones. However, these benchmarks largely ignore lexical frequency variation, despite lexical frequency being a pervasive and highly skewed property of natural language use. Consequently, existing evaluations do not test whether grammatical preferences remain stable when contrasts involve rare lexical items. We introduce FreqBLiMP, a frequency-controlled extension of BLiMP that regenerates all 67 paradigms under explicit Zipf-frequency regimes while preserving each minimal-pair's grammatical contrast. Evaluating multiple open-weight LLM families across scales, we find that decreasing lexical frequency produces a consistent, monotonic decrease in sentence likelihood, but only a modest reduction in overall contrastive acceptability accuracy. However, this aggregate stability masks substantial variation across linguistic phenomena, with LLMs remaining robust on overt morphosyntactic generalization while degrading on phenomena that require lemma-specific information.