AI peer review scores vary and depend on reviewers strongly
Can We Believe AI Peer Review?
Artificial Intelligence
Summary
The paper studies how reliable AI-generated scores are for reviewing scientific papers. The authors found that the scores can change a lot even when the same paper is reviewed again under identical conditions. Changing the scoring method can reduce how much scores vary but shifts average scores. Also, different AI reviewers give different feedback on how much a paper improves after revision.
What this means in practice
- •For conference organizing teams: Improve AI-assisted review systems by accounting for reviewer variability and scoring protocol differences to better support editorial decisions.
- •For academic journal editors: Use insights about AI score fluctuations to combine AI feedback with human reviews more effectively during manuscript evaluation and revision.
Authors
Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu
Abstract
As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.