Refverifier aids peer reviewers in checking scientific citations accurately

RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts

Software Engineering

Summary

Scientific papers often include claims supported by citations, but verifying these citations takes a lot of time, which reviewers may not have. The authors created RefVerifier, a tool that helps reviewers check if a paper’s claims match the cited sources by automatically finding and summarizing the relevant parts of cited papers. RefVerifier can find citation information, retrieve related open-access papers, and highlight evidence passages, helping reviewers make faster and more accurate judgments on citation reliability. This approach does not replace reviewers but assists them by automating some tedious steps.

What this means in practice

  • For journal editorial teams: Speed up the review process by semi-automatically verifying citation claims using a tool that links claims to evidence in cited papers.
  • For legal document reviewers: Assist in checking references and claims within legal documents by retrieving and summarizing cited evidence automatically.

Authors

Stefania Mocan, Florian Angermeir, Mark Kreitz

Abstract

As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.