Cross-pollination retrieval boosts finding scientific ideas across fields
Cross-Pollination Retrieval: Finding Transferable Ideas Across Fields in Scientific Literature
Information RetrievalArtificial IntelligenceComputation and Language
Summary
Finding ideas that apply across different areas of science is hard because papers usually stay within their own fields. The authors created a big collection of examples where papers from one research area cite useful ideas from another. They tested different ways to find these cross-field connections, showing that special training and large language models work much better than older methods. Their dataset and tools help computers better identify ideas that can be shared between scientific fields.
What this means in practice
- •For research software developers: Build search tools that find relevant papers sharing transferable ideas across different scientific disciplines using the CPR-Dataset and trained retrievers.
- •For innovation management teams: Identify promising cross-domain scientific concepts to inspire product or process innovations by accessing improved cross-field retrieval methods.
Authors
Seokwon Jung, Yerang Kim, Yejin Kim, Alexander Rubinstein, Arnas Uselis, Sangwoo Park, Seong Joon Oh, Sangdoo Yun
Abstract
Many scientific advances come from cross-pollination, carrying an idea from one field over to a problem in another. Yet retrieving papers across fields remains underexplored, with existing benchmarks confined to a narrow range of fields. We study cross-pollination retrieval, the problem of identifying transferable ideas across research fields, and introduce CPR-Dataset, a large-scale dataset constructed by mining cross-topic citations. Its evaluation split CPR-eval has 1,300 queries in 26 fields, and its training split CPR-train has 10,573 pairs. On CPR-eval, the best of eleven existing retrievers reaches a Recall@10 of only 0.26, whereas an LLM reranker reaches 0.81. Standard contrastive fine-tuning on CPR-train raises Recall@10 to as high as 0.70. The gain carries over to domains held out from training and to analogy and reasoning-intensive retrieval. On AnaloBench, it raises nDCG@10 by up to 33% over the untrained retriever, whereas training on three existing retrieval datasets lowers it. On BRIGHT, it raises nDCG@10 by up to 70%, more than twice the gain from those datasets. CPR-Dataset is available at https://huggingface.co/CPR-Dataset, and the code at https://github.com/SeokwonJung-Jay/Cross-Pollination-Retrieval.