Small language models struggle to efficiently read Indian scripts
Efficiency Measured in English: Tokeniser Coverage Bounds Small Language Models on Indic Scripts
Computation and LanguageComputers and SocietyMachine Learning
Summary
Small AI language models often report their memory use based on English, but this doesn't work for Indian scripts. The researchers found that a small model, TinyLlama, breaks Indian words into many more tokens than English words, not because it's inefficient but because its vocabulary lacks those scripts. This means the model silently falls back to reading raw bytes without showing errors, reducing effective language understanding. Other models with better vocabularies for Indian scripts perform more efficiently, showing vocabulary choice limits performance more than model size.
What this means in practice
- •For language model developers: Design tokenizers and vocabularies that better cover Indic scripts to improve small model efficiency and context length for users.
- •For machine translation teams: Build more accurate text encoders for Indian languages by avoiding silent byte fallback that inflates token counts and limits model understanding.
Authors
Anna Pramod
Abstract
Small Language Models with quantisation and parameter-efficient adaptation are widely offered as the route to multilingual AI under compute constraints, and their efficiency is almost always reported as a single memory figure. We show that this figure is measured in English and does not transfer. Across a 131k-sentence corpus spanning eleven Indic scripts, TinyLlama-1.1B's tokeniser emits 1.34 subword tokens per Latin-script word and 22.60 per Kannada word. This is not inefficiency; it is absence. Its vocabulary contains 0% of Odia's codepoints, 1% of Telugu's and 2% of Kannada's, and for five of eleven scripts the measured fertility sits at the UTF-8 byte-fallback ceiling, on Odia at 99.6% of it, with 93.6% of emitted tokens being raw bytes. Because SentencePiece byte fallback never raises an unknown-token marker, this failure is entirely silent: TinyLlama's UNK rate is 0.0% on every script, including the one it reads as bytes. We further show that low fertility can indicate the opposite of efficiency: mBERT posts its best score anywhere (1.10 tokens/word, on Odia) while discarding 87.9% of the text as UNK. MuRIL, an Indic-vocabulary encoder with a fifth of TinyLlama's parameters, reads every script at 1.23-2.26 tokens/word with zero UNK and zero byte fallback. The binding constraint is the vocabulary, not the parameter count. The consequence is concrete: on a 2048-token context window, the same model gives an English user 1,528 words of usable context and a Kannada user 91.