Multimodal language model improves clinical prognosis by integrating diverse patient data
Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis
Artificial IntelligenceComputation and Language
Summary
Clinical prognosis involves understanding many types of patient data like medical measurements, heart signals, patient history, and population trends. The authors present PRISM, a model that blends these different data types using smart number encoding, recent heart signal summaries, and a special memory of similar past patients. PRISM shows good accuracy on several clinical prediction tasks, especially when it uses data from before hospital arrival. The results suggest that combining structured health data with population context helps predict patient outcomes better.
What this means in practice
- •For hospital data teams: Integrate diverse patient data streams into unified models for improved real-time clinical risk prediction.
- •For healthcare software developers: Build clinical decision support tools that use multimodal data and retrieval techniques to enhance prognosis accuracy.$Commercial implications: Enables development of advanced prognosis software that outperforms standard models, creating market differentiation.
Authors
Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed
Abstract
Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.