Multilingual prompts cause language script confusion in large language models
Output Language Confusion under Multilingual Prompt Contamination
Computation and Language
Summary
This paper studies how large language models can get confused when given questions preceded by sentences in a different language. The researchers found that some models respond by switching their answers into the script of the distractor language, even when their original answers were correct. This script-switching can make automatic scoring systems think the answers are wrong, though many remain semantically accurate. They also found that longer English distractors cause models to often refuse to answer, showing that models can get confused by mixed-language input. The work suggests more careful evaluation methods are needed when models are used in multilingual settings.
What this means in practice
- •For multilingual ai developers: Improve evaluation of language models by separating script-switching errors from true semantic mistakes in mixed-language inputs.
- •For chatbot platform engineers: Design better fallback and response handling when chatbots receive multilingual or contaminated prompts to reduce confusion and improve reliability.
Authors
Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth
Abstract
Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingual deployment, from retrieval-augmented generation pipelines returning mixed-language passages to users pasting multilingual web content. We introduce Multilingual Distractor Interference (MDI), a lightweight and fully replicable evaluation protocol requiring no new data or annotation, in which factual questions are preceded by a semantically irrelevant foreign-language sentence, and evaluate five instruction-tuned LLMs across TruthfulQA and TriviaQA under eight distractor conditions (40,000 evaluations). Our central finding is a metric confound: for Llama-3.1-8B under a Hindi distractor, 58% of responses switch to Devanagari script, yielding a raw hallucination proxy of 0.710, but manual review reveals that 120 of 148 script-switched responses that were correct under clean conditions remain semantically correct despite being written in the wrong script, reducing the adjusted semantic hallucination rate to 0.470. All other models respond through abstention escalation with no hallucination increase. A paragraph-length English distractor triggers near-universal abstention (0.806-0.998) across all models, consistent with reading-comprehension confusion, a failure mode with direct consequences for multilingual RAG pipelines. TruthfulQA multiple-choice accuracy is unaffected under all single-sentence conditions. These results show that exact-match hallucination rates in mixed-language settings should be decomposed into script-switching and semantic error components before drawing conclusions about model reliability.