Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how pixel-based language models, which read text as images, learn languages with unique scripts like Tibetan. They looked at factors like how much training data is used, how similar the writing systems are, and whether training on related languages helps. They created new ways to measure script similarity and tested their models on several tasks. They found that models learn better when the scripts look more alike and that starting with a simpler, single-language model allows more improvement than a large multilingual model. Their work offers useful insights for adapting image-based language models to low-resource languages.
pixel-based language modelscross-lingual transferlow-resource languagesorthographic proximitycontinued pre-trainingBrahmic scriptsscript similarity metricssemantic transfermultilingual modelsmonolingual models
Authors
Ran Zhang, Miryam de Lhoneux, Wessel Poelman
Abstract
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.