Sanskrit uses more tokens than English per proposition under current tokenizers
Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition
Computation and Language
Summary
Sanskrit packs a lot of meaning into each word, so you might expect it to need fewer tokens when split up by computer programs. But the authors show that under commonly used tokenizers, Sanskrit actually needs about 1.8 to 2.9 times more English tokens for the same meaning in prose. This is mostly because Sanskrit has longer character strings rather than more tokens per character. The study also compares Sanskrit to Hindi and shows Sanskrit's tokenization cost is higher than Hindi’s, providing insights into language processing tools for Sanskrit.
What this means in practice
- •For machine translation engineers: Adjust tokenization strategies for Sanskrit to better reflect actual data needs per meaning unit in training and evaluation pipelines.
- •For language technology developers: Optimize tokenizer vocabulary sizes and settings for Sanskrit to reduce token inflation in downstream tasks such as text generation or summarization.
Authors
Devansh Sharma
Abstract
Sanskrit fuses case, number, person and tense into word endings and chains clauses into compounds, so it is information-dense per word. Whether that density survives subword tokenization is a separate question, to be asked per unit of meaning rather than per word. On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularies of 200,019 ids or more, but only 1.325-1.353 times the Hindi tokens. Against a deployed English tokenizer, Sanskrit-trained BPE arms then look cheaper per proposition than English on contemporary prose (0.887). Against a matched English control, the same algorithm and vocabulary trained on the English side of the same corpus, that flip disappears: at 32,000 and 64,000 pieces all 8 matched pairs, each size-matched arm against both a pair-matched and a byte-matched control, sit above 1.0 on prose with 95% intervals excluding it. The gap closes as the vocabulary grows: at 128,000 pieces the BPE pair reads 0.983 in domain while staying above parity out of domain (1.025) and on FLORES (1.116). The ratio factorises into a character-length ratio and a tokens-per-character ratio, the second near 1 throughout: what survives matched tokenization is character-level length, which Sanskrit prose lacks over English in SLP1 (1.028) and Sanskrit verse has (0.596). The robust statement is about deployed practice: on contemporary prose and on FLORES, with the Sanskrit side in SLP1 against the deployed o200k English pivot, Sanskrit costs 1.831-2.899 English tokens per proposition under the tokenizers people actually ship. Code, the results snapshot and every table here are public.