AI summaryⓘ
The authors studied how the way scientific papers are written, without changing their actual content, can affect how AI systems review them. They changed certain writing styles in many papers and found that some aspects, like how evidence and novelty are presented, influence AI scores more than others. The effect varies depending on the AI reviewer's initial judgment, especially in middle-range scores. More complex rewriting methods didn't consistently improve reviews, and stricter review rules lowered overall scores but didn't change how sensitive AI was to writing style. These results highlight when writing style can sway AI scientific evaluations and suggest the need for review methods that are fair regardless of phrasing.
large language modelsreward hackingrhetorical framingscientific peer reviewICLR submissionsAI evaluation protocolsmanuscript rewritingnovelty stanceevidence framingreviewer bias
Authors
Ming Li, Chenguang Wang, Xirui Li, Xinyue Zeng, Dianqi Li, Peng Shi, Dawei Zhou, Tianyi Zhou
Abstract
As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.