Pixel based language models improve prediction of natural reading patterns
Reading Time Modeling with Pixel-Based Language Models
Computation and Language
Summary
Predicting what word someone will read next helps understand how people read. The authors show that models looking directly at the word images, rather than splitting words into pieces, can better predict how long people look at words and when they skip words during reading. These models also capture visual details of letters that usual word-based models miss. This approach works in multiple languages and helps researchers study how our brains combine visual information and expectations when reading.
What this means in practice
- •For natural language processing engineers: Incorporate pixel-based predictions of word difficulty to enhance models for reading comprehension and eye tracking data analysis.
- •For user experience designers for reading apps: Use pixel-level reading difficulty estimates to adapt text presentation for smoother reading flows across different languages and scripts.
Authors
Junlin Li, Sara Alhajeri, Tatsuki Kuribayashi
Abstract
Tokenization-free, pixel-based language models (PLMs) have recently been proposed as an alternative to conventional token-based language models (TLMs), typically motivated by practical considerations, e.g., the flexibility to support new languages or vocabularies. Nevertheless, PLMs also have the potential to serve as infrastructure for computational (psycho)linguistics, where tokenization has been a source of bias in estimating information-theoretic values, such as surprisal, and has pushed the bottom-up visual orthographic features outside the scope. In this study, we first integrate PLMs into the modeling of naturalistic human reading and, in particular, examine their use to estimate next-word predictability, which can also integrate orthographic and parafoveal information. Our results demonstrate that next-word prediction difficulty, computed in pixel space, has strong predictive power for gaze duration and word skipping behavior across various languages. In particular, their advantage is complementary to existing token-level surprisal and entropy estimates. Cross-lingual analyses with script variations also clarify their distinctive visual processing biases, facilitating research into how bottom-up visual information is incrementally integrated into expectation-based reading.