Small language models adapt well to Romanized Nepali with fine tuning
Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics
Computation and LanguageArtificial Intelligence
Summary
Romanized Nepali is Nepali language written using Latin letters, which varies a lot in spelling and has few language tools. The researchers trained three small language models on examples transliterated from Nepali to Romanized Nepali to see how well they learn the language style. They found fine-tuning greatly improved the models' ability to generate text matching reference examples, but common metrics like overlap scores and likelihood disagreed on which model was best. Models with fewer tokens per word scored worse, showing tokenization matters for this task.
What this means in practice
- •For language technology developers: Develop better language models tuned for informal Romanized Nepali to improve digital communication tools in this script.
- •For multilingual nlp engineers: Fine-tune small models on transliterated data to adapt them to languages with non-standard Romanization like Romanized Nepali.
Authors
Pujan Pant, Sagar Newpane, Hari Sitaula, Manish Sakhakarmy
Abstract
Romanized Nepali, Nepali written in Latin script, is common in digital communication but has no standard spelling and few resources, and it is unclear how reliably small open language models can be adapted to it. We adapt three 2-3B base models from different families, Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, with QLoRA on 9,000 instruction examples produced by deterministic transliteration of Nepali instruction data. Each model is trained with one shared recipe and three seeds and evaluated on 1,000 held-out examples with reference-overlap metrics, byte-normalized reference likelihood (bits per byte, BPB), and an output audit, against zero-shot baselines under two prompts. Fine-tuning raises chrF++ from 10-14 to 29.0-30.6 and lowers BPB from 2.23-2.41 to 0.78-0.84; most zero-shot outputs (54-92%) are English-like. Likelihood and overlap disagree on how far apart the models are. Llama-3.2-3B has the lowest BPB in every seed, 0.042 below Qwen2.5-3B (95% CI 0.039-0.046), yet the two cannot be separated on chrF++ (30.51 and 30.61) or BLEU, even with paired item-level tests. Gemma-2-2B, whose tokenizer splits Romanized Nepali words least (2.72 tokens per word), scores worst on every measure. Which model gains most over its zero-shot baseline depends on the zero-shot prompt. In a supplementary comparison with an identical tokenizer, Llama-3.1-8B lowers BPB by 0.053 (0.050-0.056) relative to Llama-3.2-3B, while chrF++ and BLEU rise by only 0.68 and 0.58 points. Lower fertility and higher reference likelihood therefore did not identify the model whose outputs were closest to the references.