Ai reviewers improve fairness by judging scientific content not style
A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review
Computation and Language
Summary
AI systems that review scientific papers sometimes give different scores to the same research just because it is worded differently, which can unfairly reward clearer writing over better science. The authors created a large set of rewritten papers to test if AI reviewers stay fair and consistent, finding that many AI reviewers either ignore wording changes or cannot tell different papers apart. They designed a new AI reviewer called SciCore that looks at both the whole paper and its core scientific content to reduce bias from wording, performing better in fairness tests while still agreeing well with human judgments.
What this means in practice
- •For academic publishers: Improve peer-review AI by combining whole-paper and core science views to reduce bias from writing style differences.
- •For ai model developers: Develop AI reviewers with enhanced consistency in scoring papers by integrating structured scientific content into evaluations.
Authors
Chenguang Wang, Ming Li, Chengrui Fan, Jianpeng Chen, Han Chen, Tianyi Zhou, Dawei Zhou
Abstract
AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.