Benchmark predicts papers that spark new research ideas

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Artificial IntelligenceComputation and LanguageInformation Retrieval

Summary

Scientists are very good at finding old research papers that help solve new problems, a skill that AI has yet to match. To study this ability, the authors created ScholarCatalyst, a collection where lead authors marked which earlier papers helped or could have helped with their recent work, including reasons why. They tested computer programs on how well they could find these helpful papers using only information available at the time the new research began. Even advanced AI systems struggled to do better than simple search methods, pointing to the need for better training. This work aims to help build tools that can guide researchers to the right past studies when they have a new idea.

What this means in practice

Authors

Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn

Abstract

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
View PDFOpen arXiv