Pairwise expert rankings speed up discovery of promising scientific designs
Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings
Machine Learning
Summary
Finding new molecules or materials is hard because there are too many possibilities to test physically. The paper’s authors show that instead of relying on absolute measurements, researchers can use simpler pairwise comparisons—deciding which of two candidates looks better. Their method, called PRISMS, combines judgments from various experts or tools with different accuracy levels to pick the best candidates faster. This approach reduces the need for large amounts of data and speeds up finding promising designs in drug discovery and other fields.
What this means in practice
- •For drug discovery teams: Accelerate identification of promising drug candidates by integrating expert pairwise rankings for efficient experimental validation.
- •For materials design engineers: Improve selection processes for novel material designs by combining different expert evaluations without needing large datasets.
Authors
Kevin Tirta Wijaya, Alston Lo, Michael Sun, Wojciech Matusik, Vahid Babaei
Abstract
Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a validation congestion where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on curation: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ~18.8% higher hypervolume than the Bayesian optimization baseline.