BioStudyBench tests AI agents on recent biomedical study data
BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies
Artificial Intelligence
Summary
It’s hard to know if AI really understands new biomedical studies because many tests let AI use knowledge from before the studies came out. This paper introduces BioStudyBench, a new way to test AI on biomedical studies published after the AI’s knowledge cutoff date. The AI has to find the data itself, look for older publications, and analyze the data to match the study’s conclusions. The authors found that giving AI access to data and tools improves its success rate a lot, but open-weight AI models still lag behind closed-weight ones.
What this means in practice
- •For biomedical data teams: Evaluate AI tools that independently retrieve and analyze biomedical datasets to verify published study findings.
- •For clinical informatics teams: Integrate AI agents capable of automated data retrieval and analysis to assist in evidence synthesis post-publication.
Authors
David Li, Shaamil Karim, Christian Gensbigler
Abstract
We evaluate whether AI agents can match the reported findings of published biomedical studies using public data. Existing evaluations do not consistently separate analysis from prior knowledge or retrieval of the published answer. We introduce BioStudyBench, a benchmark of 25 long-horizon analysis tasks drawn from studies first published between July and September 2026, after the developer-reported knowledge cutoffs of the models we evaluate, semi-automatically filtered down from 404,019 PubMed records. In each task, the agent receives a neutral research question but no data files, so it must find and download the relevant public data, search the literature through tools that return only records dated before its cutoff, and report findings through data analysis. To measure gains over prior knowledge, we run every task both with and without access to data and tools. Across eight models, access to data and tools raises the pass rate by 47 percentage points on average over the no-data baseline. Open-weight models across sizes trail closed-weight models, with the best open-weight model passing 81.3% of tasks against 94.7% for the best closed-weight model.