Luganda language model reduces text size and improves response quality
Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX
Computation and Language
Summary
Large language models usually focus on languages with many resources, making it hard for others like Luganda to get good models. The authors built Pearl-Chat, a smaller language model specially designed for Luganda using new techniques to compress text better and understand the language's grammar. This model was trained carefully on Luganda texts and conversations, leading to much better accuracy and faster responses. They also shared the code and tools to help others use and improve it.
What this means in practice
- •For natural language software developers: Integrate Luganda-native language understanding and generation into chatbots and virtual assistants for East African users.
- •For language processing tool builders: Create tokenizers and language models optimized for agglutinative Bantu languages reducing compute and latency costs.
Authors
Wesley Kambale, Roland Sankara
Abstract
Autoregressive foundation language models predominantly target high-resource Indo-European languages, imposing computational and morphological penalties on under-represented African languages. Generic multilingual tokenizers routinely exceed fertility rates of 3.2 tokens per word on agglutinative Bantu languages, degrading context efficiency and inflating latency. We present Pearl-Chat, an 8,473,122-parameter ($\sim$8.5M) causal decoder-only transformer engineered for Luganda (Oluganda, ISO 639-2: lg), a low-resource Niger-Congo Bantu language spoken by over 10 million people in East Africa. Built from first mathematical principles in JAX and Flax NNX, Pearl-Chat uses explicit functional state decomposition (nnx.split and nnx.merge), deterministic resumable data streaming via Google Grain, and SPMD tensor parallelism using jax.sharding.Mesh. We introduce a byte-level byte-pair encoding (BPE) subword tokenizer trained from scratch on curated Luganda corpora ($V = 8,192$), achieving 4.88 characters per token compression and 1.60 tokens per word fertility, a two-fold compression improvement over LLaMA-3 (128K) and Gemma (256K). The model is trained with a two-stage curriculum: Stage 1 pre-training on 59,501 sequences across parallel, biblical, encyclopedic, and journalistic corpora, and Stage 2 instruction fine-tuning on 510 dialogues with assistant-only loss masking. Across 4,000 optimization steps, validation cross-entropy loss drops monotonically from 9.4695 (perplexity: 12,958.98) to 0.1818 (perplexity: 1.20), a decrease of 9.2877 nats. On a three-tier held-out diagnostic benchmark, Pearl-Chat achieves 0.9792 ROUGE-L and 0.9405 BLEU while maintaining concordial agreement across Bantu noun classes. We provide a Python Server-Sent Events (SSE) streaming runtime and open-source all weights, tokenizers, and code.