Large language models decide script in late layers after early reading
Script Choice in LLMs: Evidence for Late-Layer Commitment
Computation and Language
Summary
This paper studies how large language models (LLMs) handle different writing systems, or scripts, like Latin or Cyrillic. The researchers found that LLMs recognize the input and intended output scripts early on, but only fully commit to choosing which script to use in the very last layers. Interestingly, intermediate layers mostly default to Latin. Smaller LLMs are less reliable at following script instructions, suggesting deeper models handle scripts better. These findings may guide the design of future multilingual AI systems.
What this means in practice
- •For multilingual ai developers: Design deeper LLM architectures that better handle multiple scripts for multilingual tasks.
- •For natural language processing engineers: Diagnose and improve script handling by analyzing late-layer script commitment in language models.
Authors
David Kletz, Sandra Mitrović, Itay Sabato, Ljiljana Dolamić, Fabio Rinaldi
Abstract
In this paper, we investigate how script knowledge is distributed across the layers of LLMs using two complementary interpretability methods: logistic regression probing and logit-lens analysis. Our probing experiments reveal a clear asymmetry: both the input script and the instructed output script are encoded in the earliest layers of the network, while, in contrast, commitment to the actual output script emerges only in the final layers, with the model's intermediate representations defaulting to Latin throughout most of the layers. This two-stage process is confirmed by logit-lens analyses, which show that script commitment consistently occurs at the very last layers of the LLMs. Together with the weaker script-following performance observed in smaller models, these results form a converging body of evidence linking script commitment to model depth, with broader implications for the design of sufficiently deep, inclusive multilingual architectures.