Transformer encoders compared on heart signal classification tasks
Same path, different: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
Machine Learning
Summary
This paper compares two ways of building transformer models to classify heart electrical signals (ECGs). One model reuses the same layer many times, while the other stacks different layers. The models perform similarly well despite the reused-layer model having fewer parameters. However, their internal behaviors differ, affecting how they process patient data and unfamiliar signals. The study helps understand how these transformer designs work in medical signal analysis.
What this means in practice
- •For medical device developers: Improve ECG classification models by using efficient transformer designs with fewer parameters but equivalent accuracy.
- •For healthcare data teams: Deploy compact transformer models for analyzing patient ECG data with stable behavior on typical and atypical recordings.
Tested on one dataset.
Authors
Pawel Olszowiec, Michal Byra, Grzegorz Gruszczynski, Grzegorz Stefanski, Alberto Presta
Abstract
Recurrent Transformers reusing their weights rather than stacking $L$ distinct layers are becoming widely adopted due to their parameter efficiency [1,2,3]. However, the exact representational and dynamical differences between looped and stacked architectures remain uncharacterized. This paper presents a controlled study on the example of bViT model [1] applying one weight-tied block $L$ times. We train two models: bViT and standard ViT [4] on 12-lead electrocardiogram (ECG) classification tasks from the PTB-XL dataset under identical training protocols. Despite an $8.9\times$ parameter reduction, bViT achieves accuracy parity with ViT. Geometric similarity metrics demonstrate that both architectures construct comparable latent representations in an equivalent canonical order. Crucially, their dynamics differ: bViT exhibits smaller step sizes and inter-patient sensitivity, as well as near-neutral behavior away from the data manifold, whereas ViT exhibits collapsing dimensionality of representations and out-of-distribution feature expansion.