MixLID identifies languages of individual words in mixed texts
MixLID: Token-Level Language Identification in Mixed-Language Text
Computation and LanguageArtificial Intelligence
Summary
Text that mixes different languages is hard for computers to understand because they need to know which language each word belongs to. The authors created MixLID, a model that learns first on examples with just one language and then on texts that mix languages to recognize the language of each word. It works on 111 different language and writing system combinations and performs better than previous methods on tests with both single-language and mixed-language texts. The model is publicly available for others to use.
What this means in practice
- •For multilingual chatbot developers: Automatically determine the language of each word in messages mixing languages to improve response accuracy.
- •For social media monitoring teams: Identify languages of individual words in code-switched social media posts for better sentiment and trend analysis.
Authors
Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim
Abstract
Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at https://huggingface.co/nlpai-lab/MixLID.