Peer review comments split into claims checks and value judgments
ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments
Computation and LanguageHuman-Computer Interaction
Summary
Peer reviews of scientific papers often mix two different kinds of comments: checking what the paper claims and judging how valuable or suitable it is. The authors created ReviewMap, a system that sorts review comments into these two types to make it clearer. They found that large language models (LLMs) can reliably do the main sorting, but humans still need to check the details. This approach helps make review feedback easier to understand and manage for everyone involved.
What this means in practice
- •For conference organizers: Use ReviewMap classifications to create better review guidelines and dashboards facilitating clearer review discussions and rebuttals.
- •For journal editorial teams: Implement semi-automated tools that flag evaluative comment types to support human editors in analyzing reviewer feedback.
Authors
Kihyun Na, Injung Kim
Abstract
Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model $κ\geq 0.75$), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly $3\times$ as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.