Ai model learns to judge paper soundness beyond human reviews
PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
Computation and Language
Summary
Checking scientific papers involves many different feelings, but one of the most important parts is deciding if the research is done properly, which is more objective. The authors focused on training an AI to spot this 'soundness' by teaching it with many real and AI-made review comments labeled for quality. They created a big dataset from past conference papers and used this to train their AI reviewer called PaperCritic. This AI not only got better at checking conference papers but also surprisingly worked well on other types of documents like legal contracts and stories.
What this means in practice
- •For conference paper committees: Use AI to assist or semi-automate objective evaluation of paper soundness to reduce reviewer workload and improve consistency.
- •For legal document reviewers: Employ AI trained on scientific reviewing to help assess the coherence and logic of legal contracts for potential issues.
Authors
Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose
Abstract
As AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).