UMLS knowledge graphs show limited gains for medical question answering
Are We Done with UMLS Injection into Language Models for Medical Question Answering?
Artificial Intelligence
Summary
Medical language models try to answer doctors' questions using an important medical database called UMLS. The authors found that adding UMLS knowledge to these models only improves their answers a little bit. They looked closely at how researchers use UMLS and found several problems like incomplete coverage and poor retrieval methods. Based on their review, they suggest better ways to use UMLS to make medical question answering more reliable.
What this means in practice
- •For health system developers: Improve medical QA systems by addressing UMLS data coverage and retrieval issues highlighted in this paper.
- •For clinical ai engineers: Refine integration of medical knowledge graphs into language models for better diagnosis and treatment support.
A survey. It maps existing work.
Authors
Yiran Zeng, Giacomo Frisoni, Lorenzo Molfetta, Gianluca Moro
Abstract
Maybe not. The Unified Medical Language System (UMLS) has become an oracle for millions of clinicians and health system developers worldwide. Researchers have increasingly used knowledge graphs derived from UMLS to enhance language models' factual accuracy and reasoning traceability in high-stakes tasks. Medical Question Answering (MQA) is a prime testbed for integration, with benchmarks usually involving professional exercises such as disease diagnosis and drug prescription from patient history. Despite the assumed benefits, empirical evidence shows that UMLS-based graph injection typically yields modest performance gains. This paper reviews existing MQA work, critically analyzing the selected UMLS sources and their usage with language models, and probes each of these choices in isolation. We uncover methodological issues never discussed in prior work, from low UMLS coverage to weak subgraph retrieval pipelines. Based on our findings, we provide actionable recommendations to unlock the untapped potential of UMLS as an external knowledge source.