Motion guided vision language model speeds up video anomaly detection
MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Detecting unusual events in videos is important but can be slow if every frame is checked in detail. The authors created MoG-VLM, a system that uses motion to quickly decide which video parts need detailed checking by a vision-language model. This way, it spends less time on unchanging parts and focuses on suspicious actions. Their approach worked better and faster than older methods on several test sets without needing extra training.
What this means in practice
- •For video surveillance teams: Detect anomalies in live security footage more efficiently by focusing analysis on motion-driven segments instead of all frames.
- •For sports analytics developers: Identify unusual or important events in sports videos faster by prioritizing analysis around motion changes.
Authors
Khaled Waleed Dawoud, Muhammad Zaigham Zaheer, Khan Muhammad, Muhammad Haris Khan
Abstract
Vision-Language Models (VLMs) have shown strong potential for video anomaly detection, but dense frame-level inference makes them impractical for real-time deployment. Existing acceleration commonly relies on uniform sampling, which ignores content dynamics and can either waste computation on static segments or miss localized anomalies. We propose MoG-VLM a motion-adaptive anomaly detection framework that leverages motion purely as a lightweight control signal to decide when VLM inference is necessary. To this end, we introduce Temporal Gradient Magnitude (TGM) which performs content-aware sampling to concentrate computation on informative regions while avoiding redundant processing. We further introduce a hierarchical dual-stream design in which dense multi-frame VLM reasoning is activated only around frames with elevated anomaly likelihood, via anchor-triggered context-window expansion with early termination. A fusion module then reconciles these instantaneous semantic cues with temporal action dynamics. Across four benchmarks, our method achieves 91.91%, 84.30%, 87.88%, and 78.96% AUC-ROC on XD-Violence, UCF-Crime, MSAD, and ShanghaiTech, respectively, establishing new state of the art on three benchmarks under the zero-shot, training-free setting. On XD-Violence, we improve over the previous best result by +0.57% while delivering a 1.31x inference speedup. These results demonstrate that content-aware allocation of VLM capacity, processing fewer frames more intelligently rather than all frames uniformly, is a more effective axis of progress than uniformly densifying VLM inference across frames. Code is available on GitHub: https://github.com/kdawoud91/MoG-VLM.