Consolidated review training improves automated peer review quality
To Consolidate or not to Consolidate? Evaluating the Impact of Consolidation in Multi-Reference Training using Peer Reviews
Computation and Language
Summary
Generating natural language text can be simple or very open-ended, and some tasks lie in between. The authors studied generating peer reviews for papers, which have multiple ways to give feedback. They found that training computer models using a single combined version of these reviews helps the model do better than using separate, varied reviews. They created a big dataset of papers with original and merged peer reviews to show this clearly. Models trained on these combined reviews produce better outputs according to tests done by both people and automatic measures.
What this means in practice
- •For academic conference organizers: Improve automatic peer review generation by training models on consolidated multi-review data to produce higher quality review summaries.
- •For natural language processing developers: Build more effective text generation systems for tasks with multiple valid outputs by using consolidated references during model training.
Authors
Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary, Manish Shrivastava, Vasudeva Varma
Abstract
Natural language generation (NLG) tasks span the spectrum of conditional entropy, ranging from highly constrained machine translation to open-ended dialogue generation. Structured tasks like automated peer-review generation occupy the intermediate region, where a single input admits multiple valid, overlapping outputs. In this work, we demonstrate that traditional single- and multi-reference training paradigms are suboptimal for these intermediary tasks. We provide empirical evidence that consolidating diverse references into a unified training signal is crucial for developing effective systems. To facilitate this, we introduce MERC-36K, a large-scale corpus of over 36,000 papers paired with original and consolidated peer reviews. Using this dataset, we train specific architectures to isolate the impact of different reference paradigms and benchmark against existing state-of-the-art systems. Through extensive automatic and human evaluation, we demonstrate that models trained on consolidated references significantly outperform those trained on unconsolidated references. Dataset and code will be released upon acceptance.