When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
Computation and Language
Summary
The authors explore a new way to handle languages with less data that share scripts with more common languages but face problems with usual tokenization methods. They improve on byte-level models, which look at raw text characters but struggle with grouping bytes into words, especially for non-Latin scripts. Their approach links byte-level inputs to existing word-based models without needing lots of training by matching byte groups to subword embeddings and using simple grammar cues. Tests on six languages show this method helps with tasks like part-of-speech tagging, improving accuracy by up to 13.3%.
Authors
Sanjeev Kumar, Atsuki Yamaguchi, Nikolaos Aletras
Abstract
Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.