Sinhala language tool improves word splitting with neural network
A Character-Level Neural Approach to Sinhala Sandhi Splitting
Computation and Language
Summary
Sinhala language has a feature called Sandhi that merges words together, making it hard to tell where one ends and another begins. The authors developed a neural network model that works at the letter level to split these merged words back into their original parts. Their best model does well on simpler cases but struggles with more complex ones, showing room for improvement. They also found that using the native Sinhala script rather than romanized text helped accuracy.
What this means in practice
- •For natural language developers: Improve Sinhala text processing by automatically splitting merged words into their original components using character-level neural models.
- •For language technology companies: Integrate more accurate Sinhala Sandhi splitting to enhance machine translation and speech recognition systems for Sinhala speakers.$Commercial implications: This paper enables better word boundary detection, critical for commercial language services in Sinhala NLP applications.
Tested on one dataset.
Authors
Yasas Ekanayaka, Deshan Sumanathilaka
Abstract
Sinhala Sandhi splitting recovers the constituent words or morphemes hidden inside a phonologically merged surface form. The task is important for Sinhala NLP because Sandhi obscures lexical boundaries, but no prior published work has established a neural benchmark for Sinhala Sandhi splitting. We present a character-level sequence-to-sequence study based on SandhiLex, using native Sinhala Unicode input and evaluating recurrent encoder-decoder models for affixational and more complex lexicalized, derivational, and etymological Sandhi. The central challenge is the hard subset lexicalized, derivational, and etymological Sandhi, where our best model, a bidirectional LSTM encoder with a unidirectional LSTM decoder, reaches only 68.40\% exact-match accuracy (82.08\% character-level accuracy), well below the 94.00\% achieved on the more regular affixational subset. Ablations show that bidirectional encoding is the largest contributor to performance, while native Sinhala script improves exact match accuracy over romanized input. Qualitative analysis indicates that many errors are near misses involving boundary adjacent characters or plausible but incorrect phonological substitutions. These results establish an empirical baseline for Sinhala Sandhi splitting and identify data scale, Sandhi type conditioning, and attention-based decoding as the main directions for future work.