Model learns to pick video moments that explain anomalies better
Learning Temporal Evidence Selection for Video Anomaly Understanding
Computer Vision and Pattern Recognition
Summary
Understanding strange or unusual events in long videos means explaining what happens, not just spotting when something is odd. The authors show that picking which parts of the video to look at is important for good explanations. They made a model that learns to choose video moments based on objects it sees, improving the detail in explanations over just focusing on the most obviously strange parts. Their approach works better than previous methods on a big test set of annotated videos.
What this means in practice
- •For video security teams: Generate clearer explanations for unusual events in surveillance videos by selecting key visual moments linked to objects and interactions.
- •For content moderators: Improve review processes by automatically highlighting and describing suspicious or anomalous behavior in long video streams.
Authors
Hangil Park, Seungheon Kim, Tae-Kyun Kim
Abstract
Video anomaly understanding (VAU) aims to generate language-level interpretations of anomalous events in long videos, requiring visual evidence that supports explanation beyond anomaly detection alone. Existing VAU pipelines commonly rely on anomaly-centric or predefined temporal sampling strategies, which efficiently emphasize suspicious moments but overlook contextual cues, object interactions, and temporally distributed evidence needed for reasoning. Motivated by recent progress in learnable temporal selection for long-video question answering, we revisit VAU temporal sampling as a downstream-aligned evidence selection problem. We introduce the semantic-conditioned Temporal Sampling Q-Former that learns to select temporal visual evidence through the VAU generation objective. Since generic VAU prompts provide weak video-specific guidance, we condition the sampler on CLIP-embedded object tags automatically extracted from the input video. These tags are used only for temporal evidence selection and are not appended to the LLM generation prompt, separating sampler conditioning from prompt augmentation. The proposed sampler operates in visual-token space, computes soft temporal selection weights from frame-level representations, and merges duplicated selections before response generation. Experiments on HIVAU-70K show that our model achieves the best performance among the compared fine-tuned models, while ablations demonstrate that learnable temporal sampling improves over fixed anomaly-focused sampling and that CLIP-embedded object-tag conditioning further improves over generic-prompt conditioning. Our results suggest that temporal sampling in VAU should be aligned with explanation-oriented evidence utility rather than anomaly saliency alone.