Language model surprisal predicts chinese reading times differently by corpus
A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese
Computation and LanguageArtificial Intelligence
Summary
This study looked at how well language models can predict how long Mandarin Chinese speakers spend reading words. The authors dealt with challenges because the way words are split in Chinese changes depending on the system used. They found that the ability of language models to predict reading times varies depending on which text data is used and that bigger models don't always do better. Some models that are closer to traditional word statistics predicted reading times better in certain cases. Overall, the results show it’s important to test on multiple types of Chinese text to understand how language models relate to reading behavior.
What this means in practice
- •For natural language processing engineers: Evaluate language model surprisal on multiple Chinese corpora to improve reading-based applications.
- •For cognitive computing developers: Use Chinese language model surprisal to design systems that estimate human reading effort across diverse texts.
Authors
Hongao Zhu, Muxiaoqiao Xu, Yikang Liu, Siyuan Song, Yuxia Wang, Byung-Doh Oh, Hai Hu
Abstract
This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. We first propose the Shortest Matching Sequence (SMS), an alignment scheme that maps between the word segmentation assumed by eye-tracking corpora and the LMs' subword tokenization, as the two tokenizations often disagree in the context of Mandarin Chinese. Then, using a suite of Chinese-Pythia models (14M-1.4B) trained on scratch with 30B tokens, we examine how well surprisal predicts first fixation duration, gaze duration, and total reading time in three paragraph-level eye-tracking corpora of Mandarin Chinese (GECO-CN, HKP, and MECO). Contrary to previous null findings, our results show that surprisal is predictive of Chinese reading times. However, whether predictive power scales with model size and the amount of training is corpus-specific: bigger models predict better in GECO-CN, whereas inverse scaling emerges in HKP and, at the largest sizes, in MECO. Subsequently, we tested one possible explanation for the inverse scaling in HKP and found that checkpoints whose surprisal remains closer to $n$-gram statistics are better predictors of reading. All in all, the predictive power of surprisal on Chinese reading time measurements is corpus-specific, which cautions against drawing scaling conclusions from a single corpus.