Turkish English code mixing causes trouble for language AI systems
TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition
Computation and Language
Summary
Understanding messages that mix Turkish and English can confuse language technology because sometimes English words get Turkish endings, making new mixed words. The authors created a collection of over 5,000 social media posts with lots of this mixing, and marked which parts are Turkish, English, or a mix as well as named things like people or places. They tested current AI models and found these mixed words cause many mistakes, especially when finding names. The work provides helpful data and tools so others can improve AI for mixed languages like Turkish and English.
code-mixinglanguage identificationnamed entity recognitionTurkish languageEnglish languagemorphological integrationnatural language processinglow-resource languagessocial media textlarge language models
Authors
Ilayda Dogan, Phuong-Anh Nguyen-Le, Julia Mendelsohn
Abstract
Natural language processing systems underperform on code-mixed text, particularly for low-resource language pairs. Turkish-English poses a further challenge: it lets English stems combine with Turkish suffixes to form single mixed-language tokens. We introduce TurEngMix, a corpus of 5.5K noisy, naturally occurring social media posts (486,974 tokens) rich in Turkish-English code-mixing. From this corpus, we construct a new Turkish-English benchmark for code-mixed language identification (LID) and named entity recognition (NER), comprising 15K expert-annotated tokens. Evaluating both decoder LLM and fine-tuned encoder baselines, we find that monolingual Turkish and English tokens are labeled reliably, but all models have high error rates on mixed-language tokens for both LID and NER. For morphologically integrated tokens, NER error rates were 5.2x and 6.3x higher for GPT-4o and Qwen, respectively. This highlights how morphological integration remains a challenge. We release the corpus, annotations, and code to support future computational and sociolinguistic research on Turkish-English code-mixing.