HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head
2026-08-24 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created HelaBERT, two versions of a language model specifically trained on a large amount of Sinhala text. These models learn to understand Sinhala language patterns using a special tokenizer suited for its unique script and grammar. They tested the models on four different tasks like classifying news topics and identifying sentiment, showing how different model designs affect performance. The authors also introduced a new classification method that improves results on some tasks. They have shared these models publicly to help others working on Sinhala language technology.
BERTmasked language modelSinhala languageSentencePiece tokenizeragglutinative morphologytext classificationsentiment analysisnews categorizationmachine learningnatural language processing (NLP)
Authors
Thisen Ekanayake, Nisansa de Silva
Abstract
We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.