AppliedScientist improves scientific papers through AI-driven revisions
AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing
Artificial Intelligence
Summary
Improving scientific papers often depends on getting helpful reviews and then revising the work based on that feedback. The authors created a system called AppliedScientist, where an AI both reviews and revises papers in cycles. The system can fix many practical, execution-related problems in rejected papers but struggles to address big-picture ideas like novelty. This suggests AI can help make experiments and presentations better, though it is less able to improve the originality or significance of the research itself.
What this means in practice
- •For academic paper authors: Automatically improve rejected scientific manuscripts by iteratively revising experimental and implementation details based on AI-generated reviews.
- •For technical writing teams: Enhance technical reports by repeatedly assessing and refining content with AI feedback without prior bias from earlier reviews.
Authors
Vidushee Vats, Karun Sharma, Shengzhi Li, Shichao Pei
Abstract
Automated reviewing systems are increasingly evaluated based on the quality of the reviews they produce. Yet a review is only useful if acting on it leads to a measurable improvement in the paper. We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer, and evaluate it by iteratively revising rejected papers from a range of research subfields. To mirror how human authors build on earlier drafts, the AI scientist has access to its previous versions during revision. To avoid bias from prior judgments, however, each review is generated independently, with the reviewer having no memory of earlier feedback or scores. We compare three revision settings: one initialized with the original venue reviews, one initialized with AI-generated reviews, and autonomous self-revision using the same fixed prompt in every round. Because the reviewer both guides and evaluates the revision, we also assess the human-initialized revisions using Stanford Reviewer as an independent evaluator. Reviewer-guided revision consistently improves more than fixed-prompt self-revision, and Stanford Reviewer also assigns higher scores to later revisions. AppliedScientist resolves 128 of 150 execution-related weaknesses (85.3%), but only 2 of 18 idea-related weaknesses (11.1%), suggesting that iterative revision is effective at improving experiments and implementation, but rarely changes concerns about novelty or significance.