Biomedical retrieval struggles with complex ontology based constraints
OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?
Artificial Intelligence
Summary
Finding relevant medical documents using computerized methods often relies on understanding complex biomedical concepts and their relationships. The paper introduces OntologyBench, a large benchmark to test how well current search methods handle these complex queries. The authors find that simple concept matching works better than finding relations or combinations of medical traits. They also show that fine-tuning improves some results, but other advanced methods do not reliably capture the required detailed biomedical relationships. This suggests that better integration of medical knowledge and learned data representations is needed for improved search.
What this means in practice
- •For biomedical data teams: Improve biomedical search systems by integrating ontologies more explicitly to handle relational and compositional queries effectively.
- •For clinical informatics developers: Develop diagnostic support tools that better match patient phenotypes to diseases by combining learned representations with structured knowledge.
Authors
Xiao Yu Cindy Zhang, Wyeth Wasserman, Jian Zhu
Abstract
We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs across concept grounding, relational retrieval, and compositional phenotype-based retrieval. Although these tasks can be tractable using ontology-aware reference methods, across task tiers, embedding performance is generally lower on relational and compositional tasks than on concept-grounding tasks. Fine-tuning on ontology-derived supervision improves performance on several relational and compositional tasks, whereas the evaluated reranking and LLM-based candidate-scoring methods provide little or no end-to-end improvement. Errors frequently reflect diseases matching only subsets of the phenotype evidence. These findings indicate that the evaluated embedding and reranking configurations do not reliably recover the compatibility encoded by the selected ontology relations and phenotype combinations and motivate retrieval systems that better integrate learned representations with structured biomedical knowledge.