Language models reliably recognize and follow script instructions
Are Language Models Script-Aware?
Computation and Language
Summary
Language models sometimes produce text in the wrong writing system, making it hard to read. The paper explores whether these models can identify and use different scripts correctly, especially for languages that use multiple scripts. The authors found that both small and large language models can match the script of the input text and follow instructions to write in specific scripts, with larger models performing better. This ability helps ensure that computer-generated text looks as expected and is easier for people to understand.
What this means in practice
- •For software developers: Ensure multilingual apps generate text in the correct script to improve user experience.
- •For localization teams: Use language models that handle script preferences accurately when adapting content for different regions.
Authors
David Kletz, Sandra Mitrović, Ljiljana Dolamić, Fabio Rinaldi
Abstract
Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model's response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%) and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.