Physician context improves heart risk detection from chest x rays
From Image Interpretation to Clinical Reasoning: Upstream Physician-Context-Aware Multimodal Learning with Causal Reinforcement Learning
Computer Vision and Pattern Recognition
Summary
Heart attacks and other major heart problems cause many deaths worldwide. Doctors often take chest x rays to check patients, which hold clues about heart health, but current AI systems mainly focus on reading images instead of predicting future risks. The authors created a new AI method that combines chest x rays with doctors’ notes to better predict which patients might have heart problems later. Their system learns to pick out important information and think through the risks more clearly, leading to better predictions across multiple patient groups.
What this means in practice
- •For hospital data teams: Improve early identification of high-risk cardiovascular patients by integrating chest x-ray images with physician clinical notes using causal reinforcement learning.
- •For clinical decision support developers: Create tools that enhance cardiovascular risk predictions by combining imaging and clinical text with role-specific reasoning architectures.
Authors
Jialu Pi, Yanan Ma, Weijie Chen, Owen Crystal, Shubham Trivedi, Stephen Xie, Anna Silverman, Matthew Stib, Chadi Ayoub, Reza Arsanjani, Imon Banerjee
Abstract
Major adverse cardiovascular events (MACE) remain the leading cause of mortality worldwide. Opportunistic screening using routinely acquired clinical data offers a scalable approach for identifying high-risk individuals before acute events occur. Although chest X-rays (CXRs) capture latent cardiovascular biomarkers and clinical histories provide complementary patient context, existing medical vision-language models are primarily optimized for radiology interpretation rather than prognostic reasoning. We propose a causal reinforcement learning framework for multimodal clinical reasoning that integrates CXRs and physician-authored clinical histories for opportunistic MACE prediction. The framework introduces (1) a role-decoupled dual-LLM architecture that separates reasoning from risk prediction, (2) a dual-action causal reinforcement learning policy for evidence selection and reasoning optimization, and (3) causal token pruning to learn compact multimodal representations. Evaluated on an internal cohort, an emergency department cohort, and the external MIMIC dataset, the proposed framework consistently outperformed unimodal baselines and state-of-the-art medical vision-language models, achieving AUROCs of 0.720, 0.760, and 0.845, respectively. It also substantially improved reasoning quality, achieving higher GREEN scores and higher expert preference while maintaining robust predictive performance across diverse patient populations.