Multimodal transformers improve cancer survival predictions despite missing data
Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data
Machine LearningComputer Vision and Pattern Recognition
Summary
Making predictions from healthcare data is hard because information often comes from many sources and can be incomplete. The authors studied how special AI models called Transformers can combine data from different types and times, even when some pieces are missing. They found that using multiple data types together usually helps predict cancer survival better than using one type alone, but missing data causes big problems. The way the data is organized and how missing parts are handled matter a lot for reliable predictions.
What this means in practice
- •For clinical data teams: Improve cancer patient survival predictions by integrating incomplete multimodal healthcare data using optimized Transformer model designs.
- •For medical ai developers: Build more reliable AI tools for longitudinal health data by choosing effective tokenization and missing data handling for Transformer models.
Authors
Rita Cordeiro Mendes, Maria Manuel Pereira, Carlos Santiago, Catarina Barata
Abstract
Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources. However, multimodal datasets are frequently incomplete, raising important questions about how models should handle missing information. Although recent approaches increasingly rely on Transformer-based architectures for multimodal fusion, incomplete multimodal data introduces challenges such as missing modalities and intra-modal missingness, which motivate key architectural design choices, such as tokenization granularity. In addition, many datasets are longitudinal, with observations collected across multiple time points. The sequential nature of the Transformer architectures provides a natural framework for modeling such data, but introduces additional considerations, such as how temporal information should be represented. In this study, we systematically investigate these design choices using three different cancer datasets. We compare modality-level and feature-level tokenization and evaluate attention masking and imputation as strategies for handling missing information, both naturally occurring in the datasets and synthetically induced. We further investigate approaches for incorporating temporal information in our training process and how to model it. Experiments are conducted with unimodal and multimodal configurations, as well as different levels of missingness for binary survival classification and multiple time encoding strategies for discrete-time survival prediction. Our results show that while multimodal outperforms unimodal results, missing data heavily impacts predictions and thus effective design choices such as tokenization granularity, and intra-missing handling is essential for reliable prediction.