Pretraining sequence design improves clinical event prediction accuracy
Sequence Composition in EHR Foundation Model Pretraining
Machine Learning
Summary
Models that predict patient health outcomes use sequences of medical events recorded in electronic health records. Typically, all patient records are combined into one long stream for training, which can cause problems because longer patient histories get more attention and event sequences from different patients can mix. The authors show that changing how these event sequences are put together before training can improve the model’s ability to predict clinical outcomes without changing the model itself or needing more data. Their new methods work better across multiple datasets.
What this means in practice
- •For hospital data teams: Improve prediction models for patient outcomes by changing how patient event sequences are arranged during model pretraining.
- •For clinical ai developers: Enhance clinical prediction tool performance through better data sequence structuring without modifying model architecture or needing new data.
Authors
Joshua Placidi, Inigo Stibbe, Alfred J. Balston, Marek Rei, A. Aldo Faisal
Abstract
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.