Researchers weigh journal prestige and citations equally but vary widely
The illusion of neutrality in metric-based research evaluation
Digital Libraries
Summary
When deciding how good scientific work is, people often look at where it was published and how many times it’s been cited in other papers. This study asked many researchers which mattered more to them. On average, they thought both were equally important, but individuals differed a lot: some cared mostly about prestigious journals, others about citations. They also tended to value citations more than they thought their own departments did.
What this means in practice
- •For university hiring committees: Design hiring rules that account for varied preferences on prestige and citations among evaluators to balance outcomes.
- •For research policy makers: Adjust evaluation frameworks to reflect the subjective trade-offs researchers make between publication venue and impact metrics.
Authors
Fengyuan Liu, Hazem Ibrahim, Yasir Zaki, Talal Rahwan
Abstract
Venue prestige and citation counts are two widely used, albeit imperfect, signals of research quality. When the two signals conflict, evaluators must decide how much weight to assign each. Yet, it remains unknown how researchers across disciplines trade off these two signals when evaluating research outcome. To fill this gap, we surveyed 869 researchers using paired choices between hypothetical departmental hiring rules that assigned different weights to venue prestige and citation counts, asking which would produce better science. Among 795 respondents whose choices were largely internally consistent, choices placed nearly equal aggregate weight on the two signals. This apparent balance concealed substantial individual heterogeneity: nearly two in five respondents occupied the most venue-heavy or citation-heavy intervals. Moreover, respondents on average chose more citation-heavy rules than they believed their departments used in hiring. By revealing the subjective judgments that arise when research indicators conflict, our findings reinforce calls for greater caution when quantitative indicators are used to evaluate research.