Reviewers no longer reward complex words in paper evaluations
Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviews
Computation and LanguageDigital LibrariesMachine Learning
Summary
Reviewers used to give higher scores to papers that used more complex and fancy words. This study shows that over time, reviewers stopped valuing complicated vocabulary in research paper reviews, even though the papers themselves still contained them. The researchers used a clever method where an AI model rated papers consistently over many years to separate changes in papers from changes in reviewer preferences. They found that reviewers now reward other qualities like sentence variety instead of complex word use. The study helps us understand how reviewer tastes have changed independent of the actual paper content.
What this means in practice
- •For conference program chairs: Adjust reviewer guidelines to focus less on complex vocabulary and more on meaningful content qualities in paper evaluations.
- •For academic journal editors: Design editorial policies that recognize shifts in reviewer preferences to improve review consistency over time.
Authors
Jiabin Zheng
Abstract
Large language models have collapsed the cost of producing lexically elaborate prose, and whether peer reviewers still reward it is a question about the evaluator, not about the text. When the association between a writing cue and review scores moves across years, the reviewers may have changed, the submissions may have changed, or both, and a regression of scores on text cannot say which. We separate the two with a frozen rater: 81,850 machine reviews of ICLR submissions from 2018 to 2025, all generated in one February-April 2025 window with one model family and one prompt, so that its year-to-year coefficients track submission composition alone and the human-minus-frozen trend difference identifies reviewer preference drift. On 32,638 submissions with 124,615 human reviews, the human coefficient on non-domain lexical complexity falls from +0.142 to -0.015 while the frozen rater moves from +0.080 to +0.082; the three-way difference-in-differences is -0.0100 (q=0.013), and forty random-wordlist placebos through the same specification centre on zero. Humans still reward sentence-length variability, which the frozen rater never registers, while the frozen rater still pays for lexical complexity at its earlier rate. Every claim is held to a double gate of false-discovery control and interval exclusion, and the findings that failed adversarial re-testing are reported. Reviewers discounted a cue whose production cost collapsed, as models of manipulable signals prescribe; an LLM judge calibrated to historical human preferences inherits the earlier schedule and drifts out of alignment while its agreement with humans on totals stays ordinary.