Scientific taste learned from limited comparisons improves research rankings

Learning Individual Scientific Taste with Privileged Hindsight Distillation

Artificial IntelligenceMachine Learning

Summary

Different researchers prefer different types of scientific projects, even when judging by the same criteria. The paper shows how to learn these personal preferences by looking at a few examples of what each researcher likes. The method uses extra knowledge available after projects develop, but only uses information available at the time of choosing when making recommendations. Tested mainly on quantum research proposals, it reveals clear individual differences in taste and helps suggest projects that people later engage with more.

What this means in practice

  • For research funders: Make personalized research proposal rankings based on limited individual feedback to better match funders’ unique priorities.
  • For academic conference organizers: Improve reviewer assignment and paper ranking by capturing individual reviewer taste from few pairwise comparisons.

Tested on one dataset.

Authors

Hanwei Fan, Yanqiao Wang, Yiqi Zhu, Xuanyu Lei, Peng Li, Yang Liu

Abstract

Scientific taste shapes which research directions researchers pursue, and it varies across individuals. Existing work evaluates research directions along multiple evaluation criteria or models scientific taste using community feedback signals such as citations and reviews, but neither directly captures how individuals translate assessments under the same criteria into different preferences. We model scientific taste as a researcher-specific mapping from criterion-level assessments in a shared, evidence-grounded evaluation space with proposal-visible evidence, to relative preferences. A population-level mapping learned from retrospective supervision provides a cold start when individual feedback is scarce. We introduce Privileged Hindsight Distillation (PHD), which uses later-emerging scientific evidence during training to improve proposal-time criterion assessment and the population mapping, while exposing only proposal-visible information at inference. We evaluate our approach primarily in quantum research, where structured public reviews are largely unavailable, and use pairwise proposal comparison as a direct interface for observing individual taste. With only 15 comparisons per researcher, our method captures sharply different tastes, including opposing preferences over the same research-direction pairs when they trade off across evaluation criteria. In an interactive study, personalized rankings also surface proposals associated with greater subsequent user engagement, connecting learned taste to downstream research exploration.