Ai agents help check cancer study claims for reliability and errors
AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
Artificial IntelligenceComputation and Language
Summary
It can be hard and costly to check if results from scientific studies, especially medical ones, are reliable and accurate. The authors created a set of 20 cancer studies and used AI agents to try to repeat their calculations and analyses automatically. Their AI could successfully recreate most individual findings and even suggest changes to see if the results still hold. This helps identify cases where the original results might depend heavily on specific choices and flags those for experts to review.
What this means in practice
- •For medical data teams: Automatically verify and rerun analyses from cancer registry studies to identify fragile findings needing expert review.
- •For healthcare software developers: Integrate AI agents into platforms to reproduce and test robustness of published medical research findings for reliability checks.
Tested on one dataset.
Authors
Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov, Julian Killingback, Arnav Singhvi, Ronak Pradeep, Michael Bendersky, James C. Dickerson, Monica S. Lam
Abstract
Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEER-Repro, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 77.4% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline identifies potentially consequential analytical choices in 9.5% of fully reproduced findings and flags them for expert review. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.