Papers for

research grant managers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Funding proposals by average panel score finds more cited work

Funding the runners-up beats a golden ticket

Abstract: A golden ticket allows one reviewer to advance a proposal that the rest of a panel would reject. We assess its score-based rationale by comparing minority-support ranking with other rules for allocating the same number of places. We do so on 10,625 rejected submissions to a computer-science conference between 2017 and 2024, using later citation percentiles as the outcome. Minority-support ranking measured the gap between the two highest scores after adjustment for the panel mean and reviewer count. At a budget of 10% of the year's accepted count, panel-mean ranking selected submissions with a mean citation percentile 7.75 points higher than minority-support ranking (95% bootstrap interval 4.35 to 11.09). The difference was larger in the other sample periods and citation windows examined. At that budget, minority-support ranking was equivalent to a lottery within a margin of plus or minus 5 percentile points. Two further analyses, adjusted highest-score ranking and a matched comparison, found no clear citation advantage from minority support once the panel mean was accounted for. Ranking on score variance also outperformed minority-support ranking in a paired comparison. Estimates cover the 69.8% of eligible rejections whose later citations could be verified. These comparisons identify panel-mean ranking as a benchmark for evaluating whether golden tickets select work that would otherwise be overlooked.

Thu 17 SeptComputers and SocietyDigital Libraries
The gist
When deciding which rejected research proposals to fund, using a method that favors the average of all reviewers' scores tends to pick proposals that get cited more later. The authors compared this to a 'golden ticket' approach, where one reviewer can push a proposal forward even if others disagree. They found that picking proposals based on the panel's average score works better than favoring minority support or score variance. This suggests funding based on average reviewer opinion better identifies impactful research.
Open → 2609.19552v1