Multimodal memory cues aid finding scientific papers without exact details

SciToT: Multimodal Tip-of-the-Tongue Retrieval for Scientific Papers

Information Retrieval

Summary

People often struggle to find scientific papers when they remember parts of the content but not titles or author names. This problem is harder with scientific papers because memories can involve text and images like figures or charts. The authors created a big test set of scientific papers and realistic search queries that mimic how people remember. They also made computer programs to help find papers using both the text and images that people recall. Everything is shared openly to help others build better tools to rediscover scientific articles.

What this means in practice

  • For digital library developers: Improve search engines for scientific repositories by handling incomplete text and visual memory cues to help users re-find papers they partially remember.
  • For science information specialists: Assist librarians and archivists in retrieving papers when only fragmented textual or graphical information is recalled.

Authors

Stan Fris, Sijie Tao, Makoto P. Kato, Maarten de Rijke

Abstract

Researchers frequently encounter difficulties re-finding scientific papers when they remember aspects of the content but cannot recall reliable identifiers, such as titles or author names. This type of re-finding task, known as tip-of-the-tongue (ToT) retrieval, is particularly challenging for scientific documents, where recollections may involve textual information and visual elements, including figures, diagrams, and plots. Existing ToT retrieval benchmarks do not address scientific search. We introduce a multimodal benchmark for scientific ToT retrieval, comprising 108,512 open-access scientific documents across five domains and 11,499 synthetically generated queries. To construct the benchmark, we propose a scalable query elicitation pipeline that independently extracts textual and visual memory cues from source documents and uses large language models to generate queries resembling incomplete human recollections. We characterize scientific ToT information needs through an empirical analysis of publicly available re-finding requests and use the outcomes to assess the generated queries using human annotations and automated quality control. The benchmark includes lexical, dense, late-interaction, and visually grounded retrieval baselines. We release the dataset, extracted document representations, generated memory cues, evaluation splits, and accompanying generation and retrieval pipelines under an MIT license to facilitate research on multimodal scientific paper re-finding.