Llms judge generic statements similarly to humans but rely on factual truth
Why Do LLMs Accept Generics? Explaining Acceptability using Psycholinguistic and Model-Derived Factors
Computation and Language
Summary
Some statements about groups, like "birds fly," are natural for people to accept, while others feel odd. The authors studied why big language models also accept some generic statements but not others, using ideas from how humans think about these sentences. They found that the models consider how common a trait is and how well it defines the group, much like people do, but focus more on whether the statement is factually true. Changing parts of a model's internal data in directions tied to these factors changes its judgment, showing these ideas influence the model's decisions.
What this means in practice
- •For nlp engineers: Improve how language models judge and generate generic statements by tuning factors that affect statement acceptability.
- •For product content teams: Refine content quality checks by integrating model-based understanding of generic statements’ acceptability.
Authors
Mervat Abassy, Zhaoxin Feng, Tatsuki Kuribayashi, Kentaro Inui, Emmanuele Chersoni
Abstract
Psycholinguistic studies explore why humans accept some generic statements but reject others by showing that these judgments depend on more than how common a property is. People consider several other factors such as how well it sets the kind apart from others or whether it is characteristic of the kind. Inspired by the linguistic and cognitive literature, we explore to what extent factors from these studies explain how large language models (LLMs) judge generics. Our results with twelve models on a psycholinguistic dataset and on naturally occurring generics show that models weigh how common a property is against how well it distinguishes the kind, and combine the two much as people do, though less clearly. Like people, they favor characteristic properties, but unlike the accounts of human judgment, they ignore striking ones. What matters most to LLMs is whether a statement is factually true, and much of the influence of human factors reaches their ratings through that judgment. In six 7-9B models, shifting the hidden states along each factor's direction moves the judgment as the factor predicts.