Fusion of images and clinical data improves inflammatory bowel disease predictions
Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Predicting how inflammatory bowel disease patients will respond to treatment is challenging. The authors combined detailed images of tissue samples with patient clinical information using 15 different ways to merge these data types. Their best method predicted disease remission better than using images or clinical data alone, but the importance of the images varied depending on the remission type. This work helps understand how blending different medical data can improve disease outcome predictions.
What this means in practice
- •For clinical data scientists: Improve prediction models for inflammatory bowel disease remission by integrating pathology images with clinical visit data using tested fusion methods.
- •For medical image analysis teams: Design AI tools that combine histology slide information and patient data to better forecast specific remission outcomes in gastrointestinal diseases.
Tested on one dataset.
Authors
Ankana Banerjee, Mark van der Giezen, Ole Gunnar Aasprong, Emiel A. M. Janssen, Øyvind Steinsbø, Tore Grimstad, Kjersti Engan
Abstract
Multimodal AI fusion of histology and clinical data is an active research area in computational pathology, but fusion methods are rarely compared under identical conditions and across different clinical endpoints. We present a modular framework in which slide-level embeddings, obtained by training an attention-based aggregator over patch embeddings from a frozen foundation model, and per-visit clinical embeddings are fed into 15 interchangeable fusion methods with a shared output dimension and prediction head (except decision-level methods). Using a prospective cohort of newly diagnosed inflammatory bowel disease patients, we predicted STRIDE-II-aligned remission outcomes from baseline or follow-up visits. The best fusion method reached macro-F1 of 0.57-0.75, outperforming both single-modality models in four of six scenarios, and differed by outcome and input. Image reliance, the drop in balanced accuracy when the slide embedding is removed, tended to follow the remission type - highest for histological and lowest for endoscopic remission; methods with similar accuracy relied on the slide to very different degrees.