Video completion improves missing parts using structured learning and neural nets
Kronecker-Structured Video Completion with Cross-Fitted Self-Supervised Residual Learning
Computer Vision and Pattern Recognition
Summary
Videos sometimes have missing parts in certain locations over time, making them hard to repair. The authors created a two-step method where they first find a simple pattern from the seen video data, then use a small neural network to fix errors in that pattern by learning from other parts of the video. This approach keeps the original visible parts unchanged while making the missing areas look better. Their tests show it works well on both black-and-white and color videos with different missing data patterns.
What this means in practice
- •For video editing teams: Automatically fill in missing or damaged areas in videos using a learning-based method that maintains consistency with observed parts.
- •For surveillance system operators: Improve reconstruction of missing spatial-temporal video data in security camera footage for clearer analysis.
Authors
Qi Qi, Yanfei Wang, Yanjun Cui, Jun-e Feng
Abstract
Tube-wise video completion reconstructs temporal trajectories at spatial locations unobserved in every frame. We propose a two-stage framework combining regularized masked Kronecker product decomposition (KPD) with cross-fitted self-supervised residual learning. First, we estimate a compact structural backbone directly from observed entries using cyclic component-wise least-squares updates, spatial and temporal factor regularization, and regularizer-optimal scale balancing. The exact updates yield a nonincreasing objective. Second, we address the mismatch between in-sample fitting errors and the prediction errors encountered at missing tubes. We partition observed tubes into folds and construct residual targets using backbone models fitted without the corresponding target tubes. A lightweight 3-D convolutional neural network then predicts corrections from the backbone, visible residual context, observation mask, and local detail features. Exact data consistency preserves observed measurements. Experiments include ablations, sensitivity analysis, grayscale and RGB video completion, and structured missing patterns. Our method achieves the highest mean PSNR and SSIM at the higher tested sampling rates on the grayscale benchmark. On eight selected DAVIS RGB sequences, it achieves the highest mean missing-region PSNR and full-frame SSIM and the lowest mean missing-tube temporal-gradient MAE at every evaluated sampling rate.