Turkish English social media texts challenge AI language and name recognition
TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition
Computation and Language
Summary
Texts mixing Turkish and English are hard for AI systems to understand, especially when English and Turkish parts form combined words. The authors collected over 5,000 social media posts full of these mixed words and labeled the language and names within them. They tested popular AI models and found that while these models recognize pure Turkish or English words well, they make many mistakes on mixed words. This shows current AI struggles with this kind of language blending, and the authors shared their data and code to help improve future research.
What this means in practice
- •For social media analysts: Use the new labeled dataset to better detect mixed Turkish-English language usage in social media posts for sentiment or trend analysis.
- •For chatbot developers: Improve chatbot understanding and entity recognition in Turkish-English mixed conversations by training on the TurEngMix benchmark data.$Commercial implications: Enables building more accurate multilingual chatbots serving Turkish-English speakers which can be sold to global customer service platforms.
Authors
Ilayda Dogan, Phuong-Anh Nguyen-Le, Julia Mendelsohn
Abstract
Natural language processing systems underperform on code-mixed text, particularly for low-resource language pairs. Turkish-English poses a further challenge: it lets English stems combine with Turkish suffixes to form single mixed-language tokens. We introduce TurEngMix, a corpus of 5.5K noisy, naturally occurring social media posts (486,974 tokens) rich in Turkish-English code-mixing. From this corpus, we construct a new Turkish-English benchmark for code-mixed language identification (LID) and named entity recognition (NER), comprising 15K expert-annotated tokens. Evaluating both decoder LLM and fine-tuned encoder baselines, we find that monolingual Turkish and English tokens are labeled reliably, but all models have high error rates on mixed-language tokens for both LID and NER. For morphologically integrated tokens, NER error rates were 5.2x and 6.3x higher for GPT-4o and Qwen, respectively. This highlights how morphological integration remains a challenge. We release the corpus, annotations, and code to support future computational and sociolinguistic research on Turkish-English code-mixing.