Bilingual language models improve second language reading time predictions
Beetle: A Bilingual Model Suite for Modelling Second-Language Processing
Computation and Language
Summary
Understanding how people learn a second language can be confusing because many factors affect how they perform. The authors created Beetle, a system that lets researchers change one thing at a time in language models trained on two languages. They found that when these models learn languages in stages or with time-based schedules, they better predict how real second language learners read. This helps move language learning studies beyond just English and into bilingual language understanding.
What this means in practice
- •For language technology developers: Train language models with controlled data schedules to improve tools that assist bilingual users’ reading and comprehension.
- •For natural language processing teams: Develop better bilingual or second-language aware AI by using interchangeable model components and training conditions from Beetle.
Authors
Suchir Salhan, Catherine Arnett, James Michaelov, Paula Buttery
Abstract
Bilingual language models (LMs) offer a controlled setting for studying how training conditions shape second-language (L2) behaviour, but prior work typically varies exposure structure, scale, and architecture at once, making it difficult to attribute effects to any single factor. We introduce Beetle, a controlled language model pretraining framework in which tokeniser, target language, training budget, and exposure structure are each independently manipulable, enabling systematic and comparable experimentation of training conditions. Using Beetle, we train and release 285 bilingual and 45 monolingual open-source LMs with rich checkpoints across a range of exposure schedules, data scales and first languages (L1s) to study multilingual pretraining and computational modelling of bilingualism and second language learning. Evaluating models on human bilingual and second language reading-time prediction and grammaticality judgement tasks, we find that staged and temporally structured curricula consistently improve alignment with language learner reading time compared to balanced bilingual training, with the largest gains at smaller data scales and for typologically closer language pairs. The Beetle models are well suited tools to help move computational psycholinguistics beyond its prevailing monolingual, English-centric focus toward models of human bilingual processing, to study cross-lingual learning dynamics, while supporting community-based development of controlled model families.