Summary
Collecting exact experimental data from scientific papers is very slow and difficult, especially because studies report many specific details spread out in long documents. The authors created a new test called SciCurateBench that checks how well computer systems can find very precise biological experiment details from real research papers. They found that popular language models and tools struggle with this task. To help, they designed a system called LangPoli, which uses agents that write natural language instructions that other agents turn into effective actions, improving accuracy significantly. They also explored ways to teach these systems to transfer learned strategies to similar tasks faster.
What this means in practice
- •For biomedical data teams: Automatically verify detailed experimental criteria from life-science literature to speed up data curation and reduce manual checking.
- •For digital library developers: Enhance search and document ranking in biomedical databases by integrating structured experimental data extraction methods.
Abstract
Scientific research increasingly depends on curating experimental data from literature. Scientific data curation requires verifying exact experimental conditions (e.g., specific assays, clinical outcomes, and data availability) across long, heterogeneous files, which is a highly time-consuming process. Existing benchmarks evaluate only broad relevance on short queries, failing to capture this complexity. We introduce SciCurateBench, a benchmark of 200 expert-annotated papers from major life-science journals that challenges systems to verify conjunctive biological experimental criteria from raw, unprocessed scientific literature. We show that plain LLMs and popular reasoning, tool-using, and feedback-based agents all struggle with this task. To study what makes the task tractable, we further propose LangPoli, an integrative system in which a policy agent generates natural language action guidelines that actor agents convert into executable tools. LangPoli showed strong performance on both Paper Classification (83.16% F1 vs. 58.22% for its base LLM) and Structured Information Retrieval, and also improves re-ranking on the public benchmarks of LitSearch, BEIR, and TREC. We further explore curriculum transfer, a method that distills generalizable strategies from solved tasks to accelerate the solving of related problems. We hope this work facilitates research in truly practical scientific AI assistance. The anonymized pre-release can be found from https://anonymous.4open. science/r/scicuratebench-D885/.