Llm peer reviewers favor confident style over scientific content
Style over Substance? Linguistic Bias in LLM-based Peer Review Copy title
Computation and Language
Summary
Many scientific papers are now reviewed with help from large language models (LLMs). This study shows that these LLM reviewers often give better evaluations to papers that use confident and native-like English styles, even when the scientific content is the same. The authors created controlled experiments by rewriting papers to change their language style without altering the science, then had LLMs review them. They found that LLM assessments are influenced more by how the paper sounds than by what it actually says, raising questions about the fairness and reliability of AI-assisted peer review.
What this means in practice
- •For conference organizers: Improve AI tools for peer review by identifying and correcting biases toward writing style rather than content quality.
- •For academic publishers: Develop guidelines and checks to ensure AI-assisted peer review systems do not unfairly favor papers with confident or native-like language.
Authors
Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
Abstract
The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.