Large language models give lower-quality answers to non-native English users
Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models
Computation and Language
Summary
People who do not speak English as their first language often get less helpful answers from large language models (LLMs). The authors studied which aspects of non-native English cause this problem by creating a large dataset of prompts written in different English styles. They found that while simple mistakes like spelling errors don’t reduce answer quality, more complex issues like unclear organization and unusual word choices do. This means people speaking less fluent English may receive worse and less natural responses from these AI tools.
What this means in practice
- •For ai product developers: Improve language model interfaces by detecting and adapting to non-native English prompts to enhance response quality.$Commercial implications: Enables commercial AI applications that provide better user experiences for non-native English speakers worldwide.
- •For multilingual chatbot teams: Develop strategies to handle diverse English inputs by reducing disparities in language model responses caused by varied fluency.
Authors
Yusheng Zhou, Eleanor Lin, David Jurgens
Abstract
Large language models (LLMs) are increasingly used by people whose first language is not English, yet these users have been shown to receive systematically lower-quality responses than fluent speakers. Which specific features of non-native English drive this gap remains unclear, because fluency is itself a composite of mechanical accuracy, vocabulary use, organization, and discourse coherence. Here, we introduce FABLE, a controlled dataset of 190,911 English prompt variants derived from 174K real user prompts for writing-related tasks. Evaluating responses from 34 open-weight LLMs, we find a clear asymmetry; while models do not propagate surface errors such as misspellings into their outputs, models do mirror higher-level rhetorical and lexical qualities present in the user's prompt. Further, the overall quality of responses differs substantially between the least- and most-fluent prompts. These results highlight a key LLM performance disparity for non-native English LLM users, resulting in both lower-quality and less-fluent answers.