Multilingual benchmark reveals language model struggles with polarity items
BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items
Computation and Language
Summary
Some words like "ever" or "anymore" are correct only in certain parts of a sentence, usually when negation is involved. The authors created BLiMPPI, a large test set with over 12,000 pairs of such words in 33 languages to see how well language models understand these rules. They found that models have trouble especially with languages that don’t have much training data and with cases that can’t be guessed by simple word patterns. They also studied how differences in languages and grammar affect model performance.
What this means in practice
- •For natural language processing teams: Evaluate language models more thoroughly on understanding complex grammar patterns across multiple languages using BLiMPPI's large multilingual dataset.
- •For machine translation developers: Improve translation quality for rare languages by assessing and addressing model weaknesses in handling negative polarity items using BLiMPPI.
Authors
Arzu Burcu Güven, Anna Rogers, Leonie Weissweiler, Joakim Nivre, Rob van der Goot
Abstract
Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a rich, multilingual literature in theoretical linguistics, yet work on NPIs in language models is almost entirely confined to English. We introduce BLiMPPI, a multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, built by a pipeline that discovers NPIs in Universal Dependencies (UD) treebanks, labels their licensing environments, and perturbs these into minimal pairs. Evaluating 12 multilingual and 2 monolingual models alongside n-gram baselines, we find that multilingual models struggle most with lower-resource languages and with pairs that are not n-gram solvable. We further examine the effects of NPI typology, language family, licensing variability, and NPI-licenser distance. Data: https://github.com/arzuburcuguven/blimppi-data