Evaluating explanations in heart ultrasound video AI models over time
A Quantitative Evaluation Framework for Temporal Explainability in Echocardiographic Video Segmentation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Heart ultrasound videos are analyzed by AI models that try to find important parts in each frame over time. The authors found it hard to measure how well explanations of these models show consistent and meaningful changes as the video plays. They created a set of scores to check if the AI's explanations move smoothly and relate well to the heart's shape through frames. They tested different model types and found some explanations were more stable than others, but simple methods can't tell if changes mean real info or just noise. This work helps guide better tools for understanding AI in medical videos.
deep learningechocardiographyvideo segmentationtemporal explainabilityGrad-CAMConvLSTMU-Netsaliency mapsmedical imagingexplainable AI
Authors
Jiyoo Noh, Jonathan H. Chan
Abstract
Deep learning has achieved state-of-the-art performance in echocardiographic video segmentation, with an increasing number of models incorporating temporal information. However, quantitative evaluation of temporal explainability remains largely unexplored. We propose a quantitative framework for evaluating Grad-CAM explanations using four complementary metrics measuring temporal consistency, saliency motion, anatomical overlap, and temporal overlap. Using EchoNet-Dynamic, we compare a baseline 2D U-Net with ConvLSTM U-Net models trained across multiple temporal strides. While segmentation performance remained comparable across all models, intermediate ConvLSTM explanations exhibited substantially lower saliency consistency and greater centroid motion than final prediction explanations. Temporal Bottleneck explanations were significantly more stable than Encoder Bottleneck explanations across all strides, while final ConvLSTM Decoder3 explanations were broadly comparable to those of the 2D U-Net. Importantly, conventional frame-wise explanation metrics cannot determine whether variation in intermediate explanations reflects meaningful temporal feature evolution or explanation instability. These findings establish a preliminary quantitative framework for temporal explainability and motivate temporal-aware XAI methods that explicitly account for evolving representations in medical video models.