AI tools find many errors in psychology papers but miss some
How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
Computation and Language
Summary
Checking scientific papers for mistakes by hand is slow and tricky. The authors put 100 known errors into 10 psychology papers and tested several AI review tools to see how many errors they could spot. The best tool found 71 of these errors, while using all tools together caught 93. Some mistakes, especially ones where information was left out, were missed by all tools. The authors shared all their data so others can improve AI reviewing further.
What this means in practice
- •For journal editors: Use combined AI review tools to catch more errors in submitted psychology articles before publication.
- •For scientific publishing platforms: Incorporate AI review ensembles to assist human reviewers in identifying errors and omissions in manuscripts.$Commercial implications: Ensembled AI review systems could be sold as value-added services to publishers for improving review quality and efficiency.
Tested on simulated data.
Authors
Paul Litvak
Abstract
One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.