AI summaryⓘ
The authors studied how language models break down text from Indian languages into smaller parts, called tokens, and found that current tokenizers are much less efficient for these languages compared to English. This inefficiency, called a tokenizer tax, means Indian languages use many more tokens for the same content, reducing the amount that can fit in a model's context window. They showed that the problem is due to how tokenizers are designed, not the scripts themselves, and that better multilingual tokenizers can greatly reduce this issue. The authors also found that this inefficiency affects how much original content can be processed and that differences in reading comprehension are more related to language resources than tokenizer problems.
Large Language ModelsTokenizerSubword TokenizationTokenizer TaxContext WindowByte-Pair EncodingIndian LanguagesMultilingual TokenizersFLORES-200Reading Comprehension
Abstract
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as 12% of that available to English users for equivalent semantic content. We identify the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tokens, with merge failure strongly correlating with tokenizer tax (Pearson r = 0.89). We further show that this phenomenon is not an inherent property of Indic scripts but a consequence of tokenizer design. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%, demonstrating that the disparity is largely remediable. Beyond token statistics, we quantify a practical consequence by showing that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents. Finally, we examine the relationship between tokenizer fertility and reading comprehension performance on the Belebele benchmark, finding that the apparent correlation is largely explained by language resource availability rather than tokenizer behavior alone.