Video restoration improves by preserving temporal noise correlations

Taming Noise Initialization for Zero-shot Video Restoration with Image Diffusion Priors

Computer Vision and Pattern Recognition

Summary

Fixing damaged video frames is harder than fixing images because video frames are related to each other over time. The authors show that starting the fixing process with smarter noise patterns that keep these time links helps restore videos better. They use a method called DDIM inversion to create these noises based on the damaged video itself, then slowly adjust the noise to make restoration more effective. They also add a way to improve how the computer pays attention to details between frames for smoother results. Their approach works better on popular video test sets than earlier methods.

What this means in practice

  • For video post-production teams: Restore old or degraded videos by using improved noise modeling to enhance frame quality and preserve smooth motion without retraining large models.
  • For security video analysts: Improve the clarity of surveillance videos that suffer from noise and degradation by leveraging temporal correlations for better visual recovery.

Authors

Chong Wang, Zixuan Fu, Hao Cheng, Siyuan Yang, Alex C. Kot, Bihan Wen

Abstract

Recent advances in diffusion models have achieved remarkable success in restoring high-quality images in a zero-shot manner. Unfortunately, extending this success to video restoration is challenging due to the substantial computational resources required to pre-train video diffusion models. A rising trend seeks to address video restoration by utilizing a pre-trained image diffusion model, initiating restoration from either frame-wise independent or identical Gaussian noise. Yet, these approaches often fall short, as such simple noise initiations fail to capture and utilize the inherent temporal correlations within degraded video frames. In this work, we propose taming noise initialization for zero-shot video restoration, which preserves the intrinsic temporal correlations by generating initial noises directly using DDIM inversion on the corrupted video frames themselves. Building on this, we introduce an adaptive noise blending strategy that suppresses degradation patterns encoded during DDIM inversion while gradually steering the noise toward a Gaussian distribution. To further emphasize temporal consistency, we propose measurement-guided trajectory attention to better align features across frames. Extensive experiments on multiple video restoration tasks demonstrate that our method outperforms existing zero-shot approaches, consistently achieving superior results on the DAVIS, REDS4, and Set8 datasets.