Benchmark evaluates literature search tools for open research problems
Inspire: Benchmarking Scientific Literature Search for Open Research Problems
Information RetrievalArtificial Intelligence
Summary
Searching scientific papers often starts with a general problem, not a specific paper to find. The authors created INSPIRE, a test system to see how well computer programs can find useful earlier studies to help solve these problems. They test programs on real computer science topics and check how well they find and rank relevant papers from a large set. Their results show many programs struggle mostly in uncovering useful papers, and they created new data to help improve future searches. This benchmark helps improve tools that researchers might use to explore scientific knowledge better.
What this means in practice
- •For research software developers: Test and improve software tools that help researchers find relevant prior work for ongoing scientific questions.
- •For knowledge management teams: Evaluate and refine company systems that help experts discover important background papers to support innovation projects.
Authors
Jianrong Ding, Zhengyan Shi, Jianyuan Zhong, Kai Qiu, Qi Dai, Yifan Yang, Chong Luo, Qiang Xu
Abstract
Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.