Video deraining improved with semantic guided scanning method
S3VD: Semantic-Guidance Spatio-Temporal Scanning for Video Deraining
Computer Vision and Pattern Recognition
Summary
Heavy rain causes outdoor videos to become blurry and lose important details, making it harder for computers to understand the images. The paper presents a new approach called S3VD that uses smart guidance from known visual concepts and clever ways to scan video frames in space and time to clean up rainy videos. This method improves how well the video looks and keeps track of things moving in the rain better than previous techniques. The researchers show their approach works better in tests compared to earlier methods.
What this means in practice
- •For video processing teams: Enhance outdoor video clarity by reducing rain effects to improve visual data quality in surveillance and broadcasting.
- •For autonomous vehicle developers: Improve robustness of vision systems in heavy rain by better preserving spatial and temporal features in video inputs.
Authors
Kui Jiang, Yiang Chen, Yan Luo, Zhaocheng Yu, Junjun Jiang, Xianming Liu
Abstract
Heavy rainfall severely degrades outdoor videos by corrupting high-frequency details and introducing motion blur, critically undermining the reliability of visual tasks. Recently, State Space Models (SSMs), particularly Mamba, have emerged as efficient alternatives for vision tasks with their linear complexity and ability to model long-range dependencies. However, when confronted with the poor visual representations in rainy videos, Mamba still faces difficulties in preserving the integrity of 2D spatial semantics and modeling 3D spatio-temporal correlations. To break these limitations, we introduce S3VD, a Semantic-Guidance Spatio-Temporal Scanning framework for video deraining, featuring two key innovations: Multi-Scale Semantic Fusion (MSSF) Module and Spatio-Temporal Scanning Fusion (STSF) Module. The former integrates temporal semantic priors from DINOv2 to guide precise feature representation and counteract the loss of local semantic context inherent to Mamba's 1D flatten operation, enhancing robustness against extreme degradation. The latter introduces a spatio-temporal scanning mechanism and devises a Decoupled-Gating Mamba (DG-Mamba) layer, which employs two independent gates to adaptively control preceding and subsequent contextual information within the input clip, optimizing intra-frame and inter-frame correlation modeling. Experiments on video deraining benchmarks demonstrate the superiority of S3VD, achieving state-of-the-art performance with an average 0.84 dB PSNR improvement over Mamba-based baselines.