Summary
Figuring out which academic papers are good is hard and important for researchers. This study looks at different ways to use large language models (LLMs) to rate academic papers and sees how well these ratings match human reviewers' opinions and future impact like citations. The authors find that simple direct ratings by LLMs often cluster scores above acceptance thresholds and miss weaker papers, while ranking methods don't improve ordering much. The choice of LLM and how the paper is represented matter a lot, and none of the methods beat a simple rule of always rejecting papers in accuracy. Human scores still predict citations better than these LLM methods.
What this means in practice
- •For conference organizers: Improve peer review support workflows by choosing the most efficient LLM scoring methods that balance predictive agreement and API usage.
- •For journal editorial teams: Use calibrated LLM scores with abstract-only input to aid initial screening of paper submissions in large-volume review settings.
Abstract
Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.