Biomedical retrieval systems struggle with complex medical queries
OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
Artificial Intelligence
Summary
Biomedical search systems aim to find medical information by understanding complex relationships and combinations of symptoms and diseases. The authors created OntologyBench, a large set of medical queries and documents to test how well current systems work for simple and complex search tasks. They found that while some methods perform well on straightforward searches, they often fail when handling more detailed or combined medical concepts. Improving these systems requires better ways to use medical knowledge together with learned computer representations.
biomedical retrievalontologyembeddingphenotyperelational retrievalconcept groundingfine-tuninglarge language modelsreranking
Authors
Xiao Yu Cindy Zhang, Wyeth Wasserman, Jian Zhu
Abstract
We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.