AI reviewers may not notice major improvements in revised papers
Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories
Computation and Language
Summary
AI systems are used to review scientific papers, but it is unclear if they recognize when authors make big improvements after revising. The authors studied thousands of papers and their revisions, comparing these to scores given by AI reviewers. They found that AI scores often do not change much even when big improvements are made, and reviewers sometimes disagree on how scores should change. Also, a brief negative comment from a user can outweigh any positive change in scoring. This suggests that AI reviews may not consistently reflect how much a paper improves.
What this means in practice
- •For conference organizers: Improve peer review systems by understanding that AI reviewer scores may not reflect paper revisions accurately and adjust review workflows accordingly.
- •For journal editorial teams: Use insights about limited AI reviewer sensitivity to guide decisions about revisions and reviewer assignment for more reliable manuscript evaluation.
Authors
Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng, Han Chen, Kwesi Adu Cobbina, Dang Nguyen, Dianqi Li, Dawei Zhou, Tianyi Zhou
Abstract
AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.