Language models learn modal meaning better with context and training size
Modal Sense Disambiguation in Language Models: The Role of Context, Training Domain, and Input Frequency
Computation and Language
Summary
Words like 'may' and 'must' can have different meanings depending on the situation. The paper looks at whether computer language models can figure out these meanings correctly. Bigger models trained on lots of different texts do better, especially when they get extra context. Smaller models, including those trained on language kids hear, struggle more to tell the meanings apart and don’t improve much with extra context.
What this means in practice
- •For natural language processing engineers: Improve AI understanding of modal verbs by using larger models trained on varied datasets to enhance context-based disambiguation.
- •For voice assistant developers: Enhance voice assistant responses by better detecting the intended sense of modal verbs in user commands using context-aware models.
Authors
Ezgi Başar, Yevgen Matusevych, Annemarie van Dooren, Arianna Bisazza
Abstract
Modals such as may and must can take on different senses depending on discourse context, alternating between root interpretations (e.g., permission, obligation) and epistemic interpretations (e.g., knowledge, belief). While modals are known to be notoriously difficult for children to acquire, limited research exists on whether language models (LMs) can reliably distinguish between different modal senses. We evaluate seven LMs that vary in architecture, size, and training data type using vector-based similarity and surprisal measures across four datasets. Specifically, we investigate whether LMs can leverage contextual information to disambiguate modal senses and whether their performance reflects the distribution of modal senses in their training data. We show that larger LMs trained on diverse text successfully leverage additional contextual information when available. In the absence of context, they tend to favor the dominant sense based on corpus statistics. Smaller developmentally plausible LMs, including those trained on child-directed language (CDL), fail to show consistent sensitivity to modal sense distinctions and do not benefit from context to the same degree.