Policies allowing AI help barely change peer review outcomes
Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
Human-Computer Interaction
Summary
Large language models (LLMs) like AI assistants are being used in writing scientific paper reviews, but their impact is unclear. The paper reports a study at a big AI conference where some reviewers were allowed to use AI help while others were not. They found no big difference in final review scores or paper acceptance, although reviews with AI help were a bit longer. Many reviewers did not fully follow the rules about AI use. The authors discuss what this means for future review policies and tools.
What this means in practice
- •For conference organizers: Design and enforce peer review policies that realistically account for reviewer AI tool usage and compliance issues.
- •For peer review platform developers: Build review systems that detect LLM use and manage allowed versus disallowed AI assistance during review processes.
Authors
Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan, Buxin Su, Weijie Su, Alekh Agarwal, Sharon Li, Martin Jaggi, Daniel G. Goldstein, Nihar B. Shah, Miroslav Dudík
Abstract
LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investigate these questions through a randomized experiment and an anonymous post-survey at ICML 2026, a major machine learning conference involving over 24,000 papers and 17,000 reviewers. Reviewers were assigned to either a conservative policy prohibiting all LLM use or a permissive policy allowing limited assistance, with randomization among a subset of main-track papers and reviewers. Policy assignment had near-zero effects on final paper decisions, paper scores, and reviewer confidence, although reviews under the permissive policy were 5.5-7% longer. Post-survey responses (N=1,486) revealed diverse attitudes toward LLMs and substantial noncompliance: 22.5% of conservative-policy reviewers reported using an LLM despite the prohibition, and 36.5% of permissive-policy reviewers reported at least one explicitly disallowed use. We discuss implications for future peer-review policy and tool design.