Vision language models improve video anomaly detection in weakly labeled data
Parser-Free VLM Verification for Federated Weakly Supervised Video Anomaly Detection
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Detecting unusual events in surveillance videos can be difficult when data is spread across many locations, labeled only roughly, and computing resources are limited. The authors propose a method where only a small part of the system learns from all locations, while a powerful but fixed language and vision model helps check suspicious video parts afterward. They compare two ways for the model to give feedback: by generating text explanations or by providing simple numerical scores. Their tests show that the numerical score method consistently improves detection accuracy without complicated tuning, offering a simpler and more reliable approach.
video anomaly detectionvision-language modelsfederated learningmultiple instance learningweak supervisionframe-level evaluationlogit scorestext generationsurveillance datatemporal post-processing
Authors
Sébastien Thuau, Amira Gran, Siba Haidar, Rachid Chelouah
Abstract
How can vision-language models help video anomaly detection (VAD) when surveillance data remain distributed, weakly labeled, and resource-constrained? Most weakly supervised VAD methods assume centralized training; recent VLM-based extensions further rely on dense inference, generated explanations, or additional adaptation. We introduce a lightweight federated MIL-VLM cascade in which only a compact MIL scorer is trained across clients, while a frozen VLM verifies high-scoring suspect segments post hoc. We study two VLM feedback interfaces: parsed text-generation decisions and a logit-based interface that extracts a continuous anomaly score from next-token Yes/No probabilities. Experiments on UCF-Crime with InternVL3.5-2B and Qwen3-VL-2B-Instruct show that text-generation verification can improve frame-level AUC after diagnostic temporal post-processing, but remains sensitive to prompts, parsers, model choice, and smoothing. In contrast, the logit interface provides a fixed parser-free signal that improves both frame-level AUC and frame-level AP over the MIL baseline across both VLMs, without temporal post-processing in its main configuration. Since suspect segments are updated independently once available, next-token logit feedback provides a simple segment-local alternative to text-generation verification.