Papers for

product content teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Llms judge generic statements similarly to humans but rely on factual truth

Why Do LLMs Accept Generics? Explaining Acceptability using Psycholinguistic and Model-Derived Factors

Abstract: Psycholinguistic studies explore why humans accept some generic statements but reject others by showing that these judgments depend on more than how common a property is. People consider several other factors such as how well it sets the kind apart from others or whether it is characteristic of the kind. Inspired by the linguistic and cognitive literature, we explore to what extent factors from these studies explain how large language models (LLMs) judge generics. Our results with twelve models on a psycholinguistic dataset and on naturally occurring generics show that models weigh how common a property is against how well it distinguishes the kind, and combine the two much as people do, though less clearly. Like people, they favor characteristic properties, but unlike the accounts of human judgment, they ignore striking ones. What matters most to LLMs is whether a statement is factually true, and much of the influence of human factors reaches their ratings through that judgment. In six 7-9B models, shifting the hidden states along each factor's direction moves the judgment as the factor predicts.

Tue 20 OctComputation and Language
The gist
Some statements about groups, like "birds fly," are natural for people to accept, while others feel odd. The authors studied why big language models also accept some generic statements but not others, using ideas from how humans think about these sentences. They found that the models consider how common a trait is and how well it defines the group, much like people do, but focus more on whether the statement is factually true. Changing parts of a model's internal data in directions tied to these factors changes its judgment, showing these ideas influence the model's decisions.
Open → 2610.27928v1