M2G-LLM improves clinical predictions by combining text and charts
M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection
Machine Learning
Summary
Doctors use lots of different kinds of information like notes, lab results, and X-rays to understand patients. Large Language Models (LLMs) are good at reading clinical text but struggle with other types of data. The authors created M2G-LLM, a new system that connects LLMs with graph networks to mix information from different sources and also learn from patients who are similar. This helps the system make better predictions about health outcomes by looking at all the information together.
What this means in practice
- •For hospital data teams: Improve clinical outcome prediction by combining patient records, lab tests, and imaging data using integrated graph and language models.
- •For medical ai developers: Build more accurate clinical decision support tools by injecting graph-based patient similarities and temporal data into language models processing clinical text.
Authors
Inyoung Choi, Sukwon Yun, Jiayi Xin, Jie Peng, Tianlong Chen, Qi Long
Abstract
Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.