Morphology based pre-tokenization improves Nepali language modeling accuracy
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Computation and Language
Summary
When computers read Nepali words, each form with added suffixes is treated differently, which can confuse them. The authors tested a method to split Nepali words into their base parts and endings before processing, keeping important word parts intact. This approach slightly improved the computer's understanding of Nepali text in language models and helped recognize new named entities better. However, the gains for other tasks like part-of-speech tagging were small. They also provide tools and data to support future work.
What this means in practice
- •For natural language processing engineers: Improve Nepali language models by using morphology-guided token splitting to better handle word forms with affixes.
- •For speech recognition developers: Enhance recognition accuracy for Nepali named entities that include previously unseen words by applying morpheme-preserving tokenization.
Authors
Kalash Shrestha, Nikhil Pradhan
Abstract
A byte-level BPE vocabulary learns each inflected form of a Nepali word as a separate string, so a noun stem is spelled differently in each of its case-marked forms. We test whether splitting words into stem and affixes before BPE helps, with the corpus, vocabulary size, model and number of training steps held fixed. Our pre-tokenizer, Papaya, uses a finite-state transducer built from a published grammar of Nepali, falls back to regular expressions, and leaves the BPE trainer unchanged. On 607 words annotated by seven native speakers its segmenter reaches 0.96 boundary F1, and the resulting tokens keep stems intact far more often than plain BPE does. In a 17M-parameter language model it lowers bits per byte by about 1% at equal training steps; most of the larger gain seen at equal epochs comes from the extra steps that longer token sequences buy, and an unsupervised Morfessor segmentation gives the same improvement. Downstream the effect is small: NER improves only on entities that contain words unseen in training, POS tagging and news classification do not change, and published Nepali tokenizers perform about as well. We release the annotated boundary set, a 556-affix dataset and the code.