LLM framework improves error detection in scientific peer review
Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
Artificial Intelligence
Summary
As more scientific papers are published, checking their quality through peer review becomes harder and slower. The authors created a new way to help AI language models do peer reviews better by focusing on finding mistakes in papers, not just copying human reviews. They built tests by purposely adding errors to papers and developed a step-by-step review process that understands the paper well before giving feedback. This method helps AI give reviews that agree more with human reviewers and catch errors more accurately, though some weaknesses remain that need fixing.
What this means in practice
- •For conference organizers: Use the benchmark to evaluate and compare AI-assisted peer review tools that identify logical errors in submitted papers.
- •For journal editorial teams: Implement the multi-layered review framework to support reviewers in producing more thorough, efficient, and token-conscious peer reviews.
Authors
Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat
Abstract
The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.