Energy-Based Transformers as Predictors of Reading Difficulty
2026-06-22 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied a new type of language model called energy-based transformers, which connect modeling human sentence understanding to ideas from memory networks. They tested this model on reading tasks and found its energy measure predicted reading times better than traditional measures like surprisal. In a specific grammar test, this energy measure also captured known difficulties in sentence understanding. This suggests the energy measure might combine several older ways of measuring processing difficulty into one.
transformer language modelsenergy-based transformerssurprisalattention entropyreading timeassociative memoryHopfield networksrelative clausepsycholinguistics
Authors
Jakub Dotlacil, Ece Takmaz
Abstract
Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.