Language models better predict reading by tracking long stories
Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension
Computation and Language
Summary
Language models are tools that guess how humans understand language. The researchers found that when these models read only a few words at a time, they match how humans process language locally. But when they look at much longer stretches of text, they better predict how humans read overall. This is because longer context helps the models track when characters and topics are mentioned again in later sentences, similar to how people remember who and what the story is about. Disrupting this connection made the models less accurate at predicting reading behavior.
What this means in practice
- •For natural language processing engineers: Improve language model design by incorporating longer contexts to better match human text comprehension patterns during reading.
- •For ai interaction designers: Develop chatbots and assistants that maintain longer conversation histories to track entities and improve understanding of user inputs.
Authors
Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi, Ethan Gotlieb Wilcox
Abstract
Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs' context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans' reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (< 20 tokens) successfully capture local memory limitations, expanded contexts (500--1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.