Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions
2026-05-31 • Artificial Intelligence
Artificial IntelligenceHardware Architecture
AI summaryⓘ
The authors explore whether AI-generated reviews can help improve the drafting of research papers, focusing on 20 computer architecture papers. They created a tool called AI-Paper-Review that generates and organizes AI-based feedback and compares it with human peer reviews. Their findings show AI can identify many issues spotted by humans and also find some new ones, but they caution against using AI for official peer review currently. The study aims to highlight both the possibilities and limits of AI in the review process and to encourage further research.
artificial intelligencepeer reviewpaper draftingcomputer architectureAI-generated reviewethics in AIreview metricshuman-AI comparisonresearch toolsacademic publishing
Authors
Di Wu
Abstract
Research is advancing faster than ever with artificial intelligence (AI); and so are the corresponding research papers. The exploding volume of AI-generated papers have put a strain to peer review, leading to the usage of AI-generated review, potentially wide yet sneaky. However, relevant ethical concerns about confidentiality, quality, and fairness are raised and no consensus has been reached in the broad research community. We expect the debate to continue for a while, but in the meantime, we ask an alternative, practical question: \textit{can AI review improve paper drafting?} We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review, quantified by a set of metrics we define. To conduct the case study, we build a web UI-integrated tool, \emph{AI-Paper-Review}, that generates structured AI review of a draft paper, available at https://github.com/unarylab/ai-paper-review. This tool selects several AI reviewers from a diverse pool of AI reviewers and clusters and ranks their comments based on commonality and importance of review comments. It also allows to align AI comments with human comments to facilitate metric-based validation. The case study shows that AI review can cover a significant fraction of human-raised issues, but also raises issues missing in human review. This paper is not intended to encourage using AI for peer review at the current stage, but to study that (1) how AI review can improve paper drafting and (2) the potential and limitation of AI-based peer review. The release of the tool and the case study data is intended to instigate future research on this topic. Misuse for peer review would violate the ethics policies from major academic venues.