Multilingual routing benchmark tests language models across 12 languages and mixes
PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard
Computation and LanguageArtificial Intelligence
Summary
Answering questions from text in many languages is hard because most tests focus only on English. The authors created PolyRoute, a set of messages in 12 different languages and mixed-language texts to see how well models can decide routing tasks in these diverse settings. They compared several language models and found different accuracies depending on the language and mix of text. This benchmark helps measure how good language understanding is outside English using consistent tests.
What this means in practice
- •For multilingual chatbot developers: Improve chatbot message routing accuracy across 12 languages and mixed-language inputs using the PolyRoute benchmark.
- •For customer support automation teams: Use multilingual and code-mixed data to better automate routing customer messages to the correct departments internationally.
Authors
Mohammed Nayeem Khan, Sarun Kumar K S
Abstract
Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish, Arabizi), 967 optional native training rows, and a protocol for a three-way department-routing decision. Labels come from Qwen3.8-27B with a blind second pass (91% of candidates kept); an independent model, gemma4-31b, agrees on 97.2% of test items, and the largest character 5-gram similarity between any test text and any training text is 0.33. A leaderboard of 13 systems gives 61.2% for the English-root checkpoint, 77.3% for the shipped multilingual one and 96.6% for the best fine-tune; TF-IDF with logistic regression reaches 91.9%. On four withheld languages the three multilingual systems score 77%, 90% and 93%. On code-mixed text the English root (85%) is nominally ahead of the shipped multilingual checkpoint (79%; p=0.134). The split resolves differences only above about 4 points. All texts and labels are LLM-generated.