Cardiac video frames generated from sparse observations and ECG signals
Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations
Computer Vision and Pattern Recognition
Summary
Doctors often have only very limited heart data to understand how well a heart is working over time. The paper introduces CardioFAD, a method that picks a handful of key heart images that best represent the heart’s motion and uses these images aligned with heart signals (ECG) to fill in missing video frames. This helps create smooth, detailed videos of heart movement from just sparse snapshots and related signals, making it easier to assess heart function. The authors tested this on heart MRI and ultrasound videos with promising results.
What this means in practice
- •For cardiologists: Generate detailed heart motion videos from limited scans and ECG to better assess cardiac function in clinical settings.
- •For medical imaging developers: Create cardiac imaging software that reconstructs dense heart dynamics from sparse inputs using ECG-based alignment and keyframe diffusion.
Authors
Junkai Liu, Haofan Wu, Nay Aung, Joao A. C. Lima, Steffen E. Petersen, Le Zhang
Abstract
Dense cardiac dynamics are essential for assessing cardiac function, yet in practice clinicians often observe only extremely sparse measurements. However, recovering full dynamics from such sparsity is a highly ill-posed problem due to the lack of temporal cues. While auxiliary time-series signals like ECG provide physiological guidance, effectively exploiting them is challenging due to the cross-modal representation gap and phase misalignment. In this work, we present CardioFAD, a Focus--Align--Diffuse paradigm that reformulates sparse cardiac dynamic synthesis as a recoverability--alignability anchor discovery problem. Our core insight is that resolving extreme temporal ambiguity requires a compact set of anchor states that are both sufficient to recover the visual trajectory and reliable for ECG--visual alignment. We therefore learn motion-salient keyframes under two complementary principles: trajectory sufficiency, which preserves trajectory-critical intra-modal motion content, and anchor alignment, which promotes consistent inter-modal correspondence at these anchors. We implement the Focus and Align modules optimized by the two aspects. Diffuse then performs two-stage generation by first synthesizing keyframes and subsequently interpolating the remaining frames. Extensive experiments on cardiac MRI and echocardiography videos demonstrate superior visual quality, temporal coherence, and improved functional consistency under the extreme one-shot regime.