Kurate system assesses scientific paper quality with AI support
Kurate: Scalable Scientific Quality Analysis
Computation and Language
Summary
Scientific search engines find relevant papers but usually don't tell you how reliable the studies are. The authors created Kurate, a system that uses advanced language AI to judge the quality of scientific papers by looking at the paper and related documents like study protocols. Kurate scores papers on things like study design and reporting, showing where problems often happen, such as weak statistics or selective reporting. It matches expert evaluations closely and helps break down overall quality into detailed judgments linked to the paper's text.
What this means in practice
- •For clinical trial reviewers: Automatically score and report study quality on multiple criteria using linked evidence from trial protocols and publications to aid review workflows.
- •For medical journal editors: Use AI-backed quality assessments to guide editorial decisions and highlight aspects of study design needing scrutiny before publication.
Authors
Matthew J. Vowels, Jamie Cummins
Abstract
Scientific search systems can find papers that are relevant to a question, but they generally do not assess the quality of the evidence that those papers provide. We present Kurate, a system that uses large language models (LLMs) to assess the quality of published studies. Kurate uses both the paper and its related documents (e.g., the study's trial registration and protocol), and links each of its judgments to the passage of text on which that judgment is based. We applied Kurate to a corpus of 4,347 papers (3,913 of which report randomized trials) and scored each paper on 8 dimensions of study design and reporting: specifically, statistical power, causal identification, preregistration, selective reporting, measurement validity, analysis prespecification, reporting transparency, and conflict of interest and funding. Across the corpus, we found that papers most often exhibited issues with statistical power, selective reporting, and analysis prespecification, although average quality differed between clinical areas. When compared against expert annotations of 60 held-out clinical-trial documents, the information Kurate extracted matched the expert label in 221/242 protocol scorepoints and 294/370 results-publication scorepoints, with AC1 0.94 and 0.81, respectively. Using a well-reputed, high quality clinical trial as a worked example, we show how a single paper's overall grade breaks down into separate judgments, with each linked to specific evidence from the trial's registration, protocol, and published report. Together, these results show that large-scale quality assessment of this kind is feasible, and that it can be used to address meta-scientific research questions.