Knowledge graphs boost explainable cancer diagnosis from pathology images
PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology
Computer Vision and Pattern RecognitionComputation and Language
Summary
Diagnosing cancer from large digital microscope images is important but complex. The authors combine computer vision with special medical knowledge graphs to help AI systems understand and explain how they analyze these images. Their method, called PathKGen, makes AI diagnoses more accurate and easier for doctors to interpret by showing the reasoning process in clear visual graphs. This helps build AI tools that not only give cancer diagnoses but also explain their reasoning in a way doctors can trust.
What this means in practice
- •For medical imaging teams: Improve cancer diagnosis software by integrating structured pathology knowledge for more accurate and explainable reports from microscope images.
- •For clinical software developers: Enhance digital pathology tools with visible reasoning graphs that help pathologists understand AI-generated diagnoses and increase trust in automated reports.
Authors
Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe, Isiri Amani Withanawasam, Hasindri Watawana, Ranga Rodrigo
Abstract
Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in Multimodal Large Language Models (MLLMs) have enabled state-of-the-art (SOTA) automated histopathology report generation. However, MLLMs reason purely through distributions encoded in their neural parameters, limiting the amount of knowledge and reasoning logic they can contain given the model scale. Their black-box reasoning poses another challenge to the human interpretation of intermediate reasoning. We address these issues by utilizing knowledge graphs to inject external structured knowledge for MLLMs to reason over, while making this reasoning component visible. Knowledge graphs are capable of partially externalizing the containment of medical knowledge, an area that remains underexplored in pathology report generation. We introduce PathKGen, a knowledge graph-assisted MLLM for histopathology report generation. PathKGen has two main branches: (1) a visual path that learns visual token inputs to the MLLM using pathology foundation models, and (2) a knowledge path that learns knowledge token inputs by extracting key morphological features of WSIs through querying a pathology-specific knowledge graph. A comprehensive evaluation on WSI-Bench dataset shows that PathKGen surpasses previous SOTA methods while revealing diagnostic reasoning as visually intuitive graphs. Due to its modular design, the proposed knowledge path is complementary to vision-aligned MLLMs and can be extended to other applications by curating domain-specific knowledge graphs to generate knowledge tokens.