Federated learning helps reduce video sharing for surveillance anomaly detection
Federated Binary Gating with Server-Side Vision-Language Inference for Surveillance Anomaly Classification
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Surveillance systems want to detect unusual events without sharing too much private video. The authors show that splitting the work between simple local checks and a powerful server-based model that understands language helps with this. Their method flags suspicious videos locally and only sends those to the server, cutting the amount of video shared almost in half. Although this approach is less precise in some ways, changing how videos get sent can improve sensitivity to rare events. This suggests sharing information locally first is useful for privacy and efficiency in detecting problems on video.
federated learningsurveillance anomaly detectionvision-language modelsbinary gatingconvolutional neural network (CNN)zero-shot inferencenon-IID dataROC AUCmacro F1 scorevideo transmission reduction
Authors
Côme-Alexis Puech, Sébastien Thuau, Amira Gran, Arthur Mennessier, Siba Haidar, Rachid Chelouah
Abstract
Privacy-sensitive surveillance systems could benefit from large vision-language models (VLMs), but such models typically require centralized access to raw video. In federated learning settings, this challenge is amplified by non-independent and identically distributed (non-IID) client data, which can make direct multiclass anomaly classification unstable, especially for rare categories. We propose a hybrid two-stage architecture that combines a federated binary convolutional neural network (CNN) gate with server-side zero-shot VLM inference. The lightweight LiteCNN3D gate performs local anomaly screening and forwards only flagged videos to Qwen3-VL-8B, which assigns them to four anomaly metaclasses. We evaluate this design on UCF-Crime grouped into five coarse metaclasses and implement the federated stage in a real three-node heterogeneous deployment. In the studied setting, direct federated multiclass training collapses, whereas the proposed decomposition yields a better trade-off between classification quality and raw-video transmission. With fixed-threshold routing, the federated hybrid pipeline preserves nearly the same macro-averaged F1 score (F1-macro) as its centralized CNN+VLM counterpart while reducing the fraction of transmitted videos to 51.4%, although with a lower proxy macro receiver operating characteristic area under the curve (ROC AUC) than the centralized hybrid system. A complementary sensitivity-oriented routing operating point increases macro ROC AUC from 0.673 to 0.692 and reduces the false negative rate from 29.3% to 22.9%, but decreases F1-macro from 0.503 to 0.485 while increasing transmission from 51.4% to 57.9%. These results suggest that federation is better suited to coarse local screening, while routing rules can be adjusted to trade server-side VLM usage for higher anomaly sensitivity.