Human reading times reveal gaps in language model memory processing
STRUCTURALCOST: A controlled reading time dataset for modeling human sentence processing difficulty
Computation and LanguageArtificial Intelligence
Summary
People take longer to read sentences when the connection between the subject and verb is more complex, especially because of embedded phrases. The authors collected a large dataset showing this effect clearly using many people's reading times. They tested computer language models and found these models notice some difficulties but miss the extra challenge humans experience with memory. This dataset can help improve how computers understand sentences to match human thinking better.
What this means in practice
- •For natural language processing teams: Evaluate and improve language models by comparing their sentence processing difficulty predictions to human reading time data.
- •For cognitive computing engineers: Design better AI systems that integrate working memory constraints by using human reading time patterns as benchmarks.
Authors
Nina Nusbaumer, Iria de-Dios-Flores, Corentin Bel, Christophe Pallier, Guillaume Wisniewski, Benoît Crabbé
Abstract
We introduce STRUCTURALCOST, a self-paced reading dataset of 475 participants and 40,800 observations isolating the processing cost of long-distance subject-verb dependency resolution. We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Different language models -- spanning n-gram models, SSMs, and transformers -- partially mirror this graded difficulty profile, yet underestimate the integration cost humans incur, with a gap that persists across architectures and model sizes. This suggests these models capture the predictive component of human processing but not the full integration cost that working memory imposes. STRUCTURALCOST provides data needed to drive progress toward evaluating the cognitive plausibility of language models.