TraCE improves video action understanding with explainable motion concepts
TraCE: Trajectory-Aware Concepts for Explainable Video Understanding
Computer Vision and Pattern Recognition
Summary
Video understanding models often use movement information but struggle to clearly show which motions influence their decisions. The authors introduce TraCE, a method that groups motion paths, called trajectories, to identify different sources of motion like people or objects. This helps explain why the model makes certain predictions in video recognition and question answering tasks. TraCE performs nearly as well as non-explainable models while offering clearer insights into its reasoning process.
What this means in practice
- •For video system developers: Build video action recognition tools that explain predictions by linking model decisions to specific motion patterns from people and objects.
- •For interactive media designers: Create interactive video platforms with question answering that provides transparent motion-based explanations for answers.
Authors
Wooil Lee, Junpyo Hong, Jongseo Lee, Tuomas Oikarinen, Tsui-Wei Weng, Jinwoo Choi
Abstract
Video understanding models often rely on motion, yet their explanations can obscure which motion sources support a prediction. We identify motion-source under-specification as a limitation of existing video XAI: region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete. We propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts represent dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context. The resulting typed trajectory/object/scene interface supports class prediction for action recognition and question-conditioned concept matching for video question answering (VQA). We evaluate TraCE across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR. Relative to non-interpretable baselines, TraCE maintains competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA. Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging. Our project page is available at https://leewooil.github.io/TraCE_project_page/.