System improves meta-review writing by handling conflicting peer reviews
When Evidence Conflicts: Reliability-aware Meta-review Generation
Computation and Language
Summary
Writing a summary from multiple reviews is tricky when the opinions disagree. The authors created a system that checks how reliable each review is and picks the most supported points to include. This way, the summary better balances different views while highlighting strong evidence. Their tests show this works better than older methods, especially when reviews conflict a lot.
What this means in practice
- •For conference organizers: Generate balanced meta-reviews that weigh reviewer reliability when summarizing peer reviews for paper decisions.
- •For journal editorial teams: Produce clearer summary reports from multiple peer reviewers by prioritizing well-supported review opinions during editorial decision processes.
Authors
Xinzhe Wang, Fei Tao, Jiang Xie, Hong Yu, Ye Wang
Abstract
Generating coherent meta-reviews from multiple peer reviews is challenging when reviewer evidence conflicts and varies in reliability. Existing approaches typically formulate meta-review generation as a multi-document summarization task and aggregate reviewer feedback uniformly, making it difficult to determine which opinions should be prioritized under disagreement. In this paper, we study meta-review generation through reliability-aware evidence aggregation. Our framework first extracts aspect-level opinions from peer reviews and identifies conflicting evidence within each aspect. It then estimates opinion-level support and review-level quality to measure evidence reliability. Based on these signals, the framework assigns reliability-aware weights to reviewer feedback, enabling the generator to prioritize better-supported arguments while preserving diverse perspectives. Experiments demonstrate that our method consistently improves meta-review generation over strong baselines on both automatic and human evaluations, with clear gains in conflict recognition and resolution under high-conflict review scenarios. The code and implementation details are publicly available at https://github.com/Wangxz729/reliability-aware-meta-review.