Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Software EngineeringArtificial Intelligence
Summary
The authors studied how different natural languages used to describe programming tasks affect the code generated by large language models like GPT-4o mini, DeepSeek, and Claude. They tested 460 tasks in Python and Java, translating the prompts into Chinese, Hindi, Spanish, and Italian while keeping the meaning the same. Their results show that English prompts are not always the best for generating correct or high-quality code, and the impact varies depending on the programming language and model. They also found that generated code often mixes English with the prompt language in comments and strings. This work provides a new multilingual benchmark to better understand and improve code generation across different languages.
Authors
Saima Afrin, Alessandro Midolo, Camilo Escobar-Velásquez, Mario Linares-Vásquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo
Abstract
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.