Concept bottleneck models improve video recognition with coarse concepts
Drop the Details, Get the Gist: Dynamic Conceptual Representation for Video Recognition
Computer Vision and Pattern Recognition
Summary
Video recognition can be made more understandable by using human-friendly concepts to explain how a machine identifies actions. The authors extend this idea from images to videos by breaking videos into segments and representing each segment with scores for different concepts. These scores are analyzed over time using a transformer model to recognize the video’s overall activity. They found that using simple, broad concepts works well enough for understanding cooking and household videos, so detailed descriptions aren’t always needed.
What this means in practice
- •For video analytics teams: Improve activity recognition systems by using coarse concept sequences and transformer models for better video classification interpretability.
- •For home automation developers: Develop smarter home activity monitors that classify household tasks using simpler, human-understandable concepts over time.
Authors
Ryoya Yoshiyama, Bowen Wang, Yuta Nakashima
Abstract
Concept Bottleneck Models (CBMs), which route predictions through a layer of human-interpretable concepts, have substantially improved the interpretability of visual classification by exposing an explicit, language-based rationale for each prediction. Most existing work focuses on static images, and temporal dynamics in videos have been underexplored in conceptual representation. We propose a framework that extends concept-based image recognition to procedural video understanding. It represents a multi-step video as a temporal sequence of segment-level concept score vectors and feeds this sequence into a Transformer encoder to predict the class label. We also explore a way to estimate the contribution of each concept for further interpretability. We evaluated three concept sets of different granularities on cooking and household-activity videos and found that coarser-grained concepts are sufficient for classifying these videos, which implies that fine-grained textual descriptions may be unnecessary.