Large language model judge choice matters more than prompt style
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Evaluating language tasks with large language models (LLMs) involves giving the model specific instructions or prompts. This study finds that which LLM is used as a judge influences evaluation results more than the style of prompting itself. The effectiveness of prompting strategies varies a lot across different languages, models, and tasks. It shows that evaluation methods that work well for English may not transfer well to other languages and judges. The authors suggest testing prompting strategies specifically for each evaluation setup instead of assuming a one-size-fits-all approach.
What this means in practice
- •For multilingual nlp engineers: Tune evaluation prompts specifically for the LLM judge and languages used in your tasks to get more accurate performance measurements.
- •For language technology product teams: Design evaluation workflows that consider judge model selection to improve reliability across languages and task types.
Authors
Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
Abstract
LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.