Model conditioned keyframe extraction improves video explanation clarity
MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation
Computer Vision and Pattern Recognition
Summary
Videos are often long and complex, making it hard to understand what a computer model thinks is important. The authors developed a way to pick just a few important frames based on what a video model predicts, keeping the model’s belief intact. They also found a fair way to assign credit to each keyframe for how much it influenced the model’s output. This approach helps explain video models better than looking at individual frames alone. They also began exploring how to highlight changes between keyframes to further clarify important video moments.
What this means in practice
- •For video analytics developers: Generate concise, model-relevant video summaries that preserve classifier decisions for easier debugging and interpretation.
- •For machine learning platform engineers: Integrate model-conditioned keyframe attribution to enhance transparency features in video understanding systems at scale.
Authors
Lahiru Wijayasingha
Abstract
We show that a video can be represented with just a few keyframes without sacrificing a given model's belief about the video. Our method is the first of its kind that extracts these keyframes conditioned on the model's prediction, which is massively helpful for building explainable video models. We then fairly distribute the importance of the video to the model among the keyframes with game theoretically fair Shapley values. Through qualitative and quantitative experiments, we show that keyframe attribution is superior to the traditional frame attribution. Further, we conduct preliminary exploration on highlighting the important changes among the frames by looking at the differences between adjacent keyframes.