Phonemic factorization improves chinese and vietnamese language models
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Computation and Language
Summary
Language models usually break words into characters or smaller word parts but miss the internal sounds inside syllables. The authors create a new method that breaks syllables into three sound parts—onset, rime, and tone—based on their actual pronunciation, which keeps the sequence length the same but makes the model understand related sounds better. This method uses very small vocabularies and works for Vietnamese and Chinese. Their language model built this way matches or beats standard approaches on language tasks in both languages.
What this means in practice
- •For natural language processing engineers: Build more compact and efficient pretrained models for Vietnamese and Chinese text using phoneme-based tokenization.
- •For speech recognition developers: Use factorized phonemic representations to improve handling of tonal and syllabic structures in multilingual speech-to-text systems.
Authors
Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le, Phu Gia Hoang, Dat Tien Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen
Abstract
Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies. We introduce \textbf{Phonemic Tokenizer}, a linguistically motivated tokenizer for Vietnamese and Chinese that converts each syllable into IPA and factorizes it into three phonological components: onset, rime, and tone. The three components jointly occupy one contextual position, preserving syllable-level sequence length while enabling representation sharing across phonologically related syllables. Non-phonological and unsupported units are handled through character-level fallback. This deterministic design requires no corpus-dependent vocabulary learning and yields vocabularies of only 112 entries for Chinese and 256 for Vietnamese. Intrinsic evaluation shows that the tokenizer achieves substantially higher Rényi efficiency in both languages, represents every entry in a standard Vietnamese syllable dictionary with a Fertility of exactly one, and generally produces shorter Vietnamese sequences than existing pretrained tokenizers. We further instantiate the tokenizer in \textbf{PhonemicBERT}, which combines factorized component embeddings and reconstructs complete masked syllables using three prediction heads. Under a controlled Chinese pretraining setup, PhonemicBERT-Zh is competitive with or outperforms character, subword, and SubChar alternatives across diverse language-understanding tasks. PhonemicBERT-Vi also achieves competitive or superior results to established Vietnamese and multilingual pretrained models. These results establish phonemic factorization as a compact, efficient, and interpretable alternative to atomic and statistically segmented text representations.