ReviewPilot automates paper screening and data extraction in reviews
ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
Computation and LanguageArtificial IntelligenceInformation Retrieval
Summary
Systematic reviews check many scientific papers to decide which to include and to gather important data. The authors created ReviewPilot, which uses multiple AI agents to help researchers quickly screen papers and extract information with clear reasons and sources. Unlike simpler tools, ReviewPilot remembers user settings and explains its decisions. Tests showed it works better than some existing AI methods, especially for health studies, and works well in other fields too. The tool and its code are available for anyone to use.
What this means in practice
- •For medical review teams: Help health researchers automatically screen papers and extract study data with clear tracking of decisions and sources.
- •For human-computer interaction teams: Assist HCI teams in processing relevant literature by automating inclusion decisions and extracting key data while keeping records of sources.
Authors
Zhaoqian Xue, Shaoze Zhou, Xuexin Li, Junting Mu, Haozheng Liu, Zhongyin Wang, Yichun He, Renkai Ma, Hao Zhou, Sarah Bratt, Siyuan Ma, Chen Chen, Lingyao Li
Abstract
Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.