Multimodal video recovery improves corrupted video quality with less memory
Reference-Efficient Multimodal Bitstream-Corrupted Video Recovery via Depth-Conditioned Adaptation
Computer Vision and Pattern Recognition
Summary
Video data can get damaged during storage or transmission, which makes the videos look bad or incomplete. The authors focus on improving video recovery using both color images and depth information to fill in missing or corrupted parts. Their method picks one high-quality reference image to guide the repair, making the process more efficient and producing better quality videos. This approach also reduces the computing power and memory needed during recovery.
What this means in practice
- •For video streaming engineers: Restore corrupted video streams using depth information while reducing memory and latency demands on receivers.
- •For multimedia storage providers: Improve retrieval quality of damaged video files by efficiently leveraging 3D geometry alongside color references during decoding.
Authors
Tianyi Liu, Jiaojiao Wang, Ye Lu, Kejun Wu, Chengtao Cai, Zhengguo Li, Chia-Wen Lin
Abstract
Recovering visually faithful content from damaged video bitstreams is important for reliable multimedia communication and storage. Existing methods exploit local residuals and non-local RGB references of the input video sequence, but both sources can become unreliable after decoding, while processing additional references increases receiver computation and memory. This paper studies multimodal bitstream-corrupted video recovery (MBSCVR) using RGB and depth information, where pre-transmission depth-based geometry aids reconstruction when the received appearance is corrupted. We propose a novel MBSCVR framework that selects and effectively utilizes a single selected non-local RGB reference per recovery window to achieve higher performance. It integrates a reliability-aware multimodal selector that combines valid appearance with depth compatibility before reference encoding. Depth-Conditioned Reference Adaptation (DCRA) introduces selected RGB evidence into feature completion and content generation, while frame-aligned depth conditions completion queries. On the complete BSCV benchmark with oracle masks, MBSCVR uses one selected reference to achieve 32.30 dB on YouTube-VOS and 28.93 dB on DAVIS, surpassing full-reference B2SCVR by 0.27 and 0.33 dB and improving SSIM on both subsets. Matched receiver-forward measurements show 36.8\% lower latency and 40.8\% lower peak memory. These results demonstrate how compact geometric guidance and adapted appearance processing improve the quality--resource trade-off of multimedia recovery.