GPT models generate more diverse edit questions for AI papers
Generating Edit-Inducing Questions for AI Research Manuscripts
Computation and Language
Summary
Improving scientific papers can be guided by asking helpful questions that encourage edits. This study looks at how well AI language models, like GPT, can make such questions compared to human reviewers. The authors found that GPT asked more questions that lead to changes and covered more parts of the paper. However, fewer of GPT’s questions actually resulted in edits. The study also showed that giving GPT the full paper context can sometimes reduce the quality of its questions.
What this means in practice
- •For academic paper authors: Provide automated questions to authors that prompt more extensive and broader revisions in AI research manuscripts.
- •For scientific conference editors: Incorporate AI-generated questions to complement human reviews and improve the quality of paper revisions.
Tested on one dataset.
Authors
Sebastian Joseph, Zichao Wang, Jennifer Healey, Alexa Siu, Junyi Jessy Li, Ani Nenkova
Abstract
We study the ability of LLMs to generate edit-inducing questions whose answer will improve a paper draft. On a dataset of paired submission and camera-ready papers from ICLR and NeurIPS, we compare the helpfulness of questions from GPT models with or without full paper context to that of human reviewers. GPT produces more edit-inducing questions and its questions are associated with more extensive edits and cover a broader range of edited content compared to questions from reviewers. However, a much smaller percentage of the GPT questions are edit-inducing. Our analyses confirm that automated questions can be beneficial to authors and highlight an example task where proper attending to long context deteriorates reasoning model ability to produce helpful output.