Language identification models improve tri-language social media texts
IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Computation and Language
Summary
People often mix three languages in a single sentence on social media, making it hard for computers to know which language each word belongs to. The authors worked on solving this by teaching language models to recognize languages at the word level for mixed sentences involving Hindi, Gujarati, and Bengali. They created and tested new datasets and showed that advanced transformer-based models help identify code-mixed languages better. Their work also includes sharing datasets and models for others to use and build on.
What this means in practice
- •For social media platform developers: Integrate token-level language identifiers to improve moderation and content personalization for users mixing Hindi, Gujarati, and Bengali languages.
- •For multilingual chatbot builders: Use fine-tuned language identification models to enable chatbots to understand and respond accurately to code-mixed input with three languages.
Authors
Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana, Shrikant Malviya
Abstract
Language identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity. Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings. We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages. We evaluate these systems on three different data configurations (Hindi, Gujarati, and Bengali) to predict language labels for individual tokens. We release a benchmark for language identification in code-mixed tokens with manually annotated test sets. We propose two approaches of code-mixed generation using parallel sentences of three languages. The trained models demonstrate the effectiveness of contextual embeddings for token-level language identification in multilingual social media text. For reproducibility and to facilitate future research, we publicly release our fine-tuned models.