Reading selective scan reveals key video parts driving predictions
READ: Reading the Selective Scan for Faithful Video Attribution
Computer Vision and Pattern Recognition
Summary
Selective state-space models help computers understand long videos efficiently, but it is hard to tell which parts of the video influence their decisions. The authors designed READ, a new way to trace back a computer’s decision to specific video frames and regions without needing extra heavy calculations. This method shows clearly which parts of the video mattered most for the prediction, using only one pass through the data. Their tests show READ matches or beats more complex methods, making video analysis more transparent.
What this means in practice
- •For video analytics engineers: Identify which video frames and regions influence automated predictions efficiently for model debugging and improvement.
- •For security monitoring teams: Reveal meaningful video segments that led to flagged events using a low-cost attribution method to support incident review.
Authors
Magdalena Trędowicz, Piotr Kubaty, Arkadiusz Lewicki, Jacek Tabor, Łukasz Struski
Abstract
Selective state-space models are an efficient approach to long-sequence video modeling. However, their recurrent computation makes it difficult to determine which video tokens drive a prediction. Existing attribution methods often require additional backward passes or repeated model evaluations, making them substantially more expensive than the forward inference they aim to explain. In this work, we introduce READ, a parameter-free attribution method for selective state-space models that unrolls the recurrence at the position read by the classifier. READ traces the contribution accumulated in the recurrent state back to the video tokens that contribute to the final prediction. By anchoring attribution at the classifier's readout, it produces a single spatiotemporal relevance map from which frame- and region-level relevance can be obtained directly. Despite requiring only one forward pass, READ achieves competitive faithfulness and localisation compared with substantially more expensive attribution methods. Experimental results show that READ avoids the systematic relevance biases of existing methods while recovering meaningful temporal and class-specific structure across datasets and backbones. More broadly, our results show that the recurrence underlying selective state-space models is not only an efficient mechanism for inference, but also a direct source of interpretable evidence for their predictions.