Adaptive method improves multi-label video harmful content detection
Controllable Multi-label Video Safety Detection via Adaptive Tversky Policy Optimization
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Videos shared online sometimes contain harmful content, and spotting all types of unsafe material is tricky because a single video can have multiple problems. Current systems often just say yes or no to whether a video is unsafe, missing this complexity. The authors developed a new technique that can detect many types of harm at once and can be adjusted to focus more on catching unsafe content or avoiding false alarms. Their tests showed this method works much better on real-world video safety challenges and can be tailored for different moderation needs.
What this means in practice
- •For social media moderation teams: Adjust video safety filters to balance catching harmful content versus avoiding false warnings based on specific platform policies.
- •For online content platform operators: Improve automated detection of various harmful video categories to enhance user safety across diverse user-generated videos.
Authors
Guangyu Yang, Jingbiao Mei, Mingsheng Sun, Jinghong Chen, Yingtong Bu, Pengda Qin, Da Chen, Bill Byrne
Abstract
The rapid growth of video-based social media has increased users' exposure to harmful content, creating a need for reliable automated video safety detection. Although recent Vision-Language Models (VLMs) show strong video understanding capabilities, existing harmful video detection systems face two key limitations: they typically reduce safety detection to binary classification, overlooking the inherently multi-label nature of unsafe videos, and they rely on static training objectives that do not support controllable precision-recall trade-offs, though the desired operating point may vary across moderation pipelines and unsafe categories. To address these gaps, we propose Adaptive Tversky Policy Optimization (ATPO), a reinforcement learning framework for Multi-label Video Safety Detection (Multi-VSD). ATPO introduces the Adaptive Tversky Reward (ATR), which dynamically adjusts false-positive and false-negative penalties during training to enable controllable precision-recall trade-offs. Experiments on SafeWatch-Bench and XD-Violence show that ATPO substantially improves multi-label performance, increasing the Jaccard Index from 40.66 to 75.44 on SafeWatch-Bench-Real. Moreover, ATR enables reliable steering of the precision-recall operating point, supporting deployment scenarios with heterogeneous policy requirements. Code and checkpoints are provided at https://bruceyg.github.io/ATPO-project-page/ .