Premisecheck speeds fact checking of human and ai peer reviews
PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Review comments on scientific papers sometimes mistakenly say things the paper doesn’t actually say. The authors introduce PremiseCheck, a method that breaks down these review comments into clear statements and checks each one against the paper using a fast decision model. This approach is almost as accurate as a large language model that reads the whole paper but is cheaper and faster. It works well across fields like natural language processing and natural sciences. This helps make peer review more reliable and efficient.
What this means in practice
- •For conference organizers: Improve reliability of peer review by automatically checking factual accuracy of human or AI-written review comments.
- •For journal editorial teams: Speed up the editorial process by quickly flagging factual errors in reviewer comments to focus human attention efficiently.
Authors
Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun
Abstract
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.