Peer review funnel redesign drastically cuts workload and improves top paper selection
Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood
Machine LearningComputers and SocietyDigital Libraries
Summary
As more papers flood into big conferences like NeurIPS, reviewers spend many years worth of time reading most papers that get rejected. The authors show that simply limiting who can submit does not help much and unfairly hurts new authors. Instead, they propose a multi-stage review process where an automated check filters out the weakest papers first, then many readers rank the rest from short summaries, and only the finalists get full reviews. This approach reduces required expert hours by about half while accepting more top-quality papers and highlighting the very best consistently.
What this means in practice
- •For conference organizers: Implement a staged funnel with automated checks and partial reads to reduce reviewer time and improve acceptance of high-quality submissions.
- •For journal editorial teams: Adopt multi-phase paper ranking methods using initial automated screenings and partial evaluations before deep reviews to handle increasing submission volumes.
Authors
Mikhail L. Arbuzov, Lee Mosbacker, Sisong Bei
Abstract
Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.