People use new verbs flexibly but AI models depend on meaning
Examining form-based linguistic productivity in people and large language models
Computation and Language
Summary
People can use new made-up verbs in different sentence types even when the verbs have no clear meaning, showing they understand abstract language patterns. The authors show that large language models, unlike people, only manage this when the new verbs are similar in meaning to ones they already know. This suggests that AI models link grammar closely to word meanings, while humans use more flexible rules about how words fit together.
What this means in practice
- •For natural language processing engineers: Improve language models by focusing on form-based generalization beyond word meanings to better handle novel language use.
- •For computational linguists: Develop better syntactic evaluation benchmarks by testing models’ ability to apply abstract grammatical rules in unfamiliar contexts.
Authors
Michael Wilson, Robert Frank
Abstract
Linguistic productivity, as evidenced by the use of words in novel contexts, is one of the most striking facts about human language. However, how abstract the knowledge that enables such productivity is remains unclear. We compare people's and computational language models' linguistic productivity in contexts where successful generalization depends on form-based information. We focus on extending the use of a novel verb learned in the active voice to passive contexts. We find people extend active to passive uses in a way consistent with abstract generalizations concerning linguistic form, even when no plausible meaning can be assigned to the novel verb. In contrast, large language models succeeded only when a meaning like an existing verb's could be assigned to the novel verb. We conclude that computational models' syntactic knowledge remains tied to the distribution and meaning of specific lexical items in a way that human syntactic knowledge is not.