Efficient image restoration with flexible learned degradation operators
Perturb-and-Solve: Efficient Learned-Operator Conditioning for Latent Diffusion Inverse Problems
Machine LearningComputer Vision and Pattern Recognition
Summary
Restoring images that are blurry, missing parts, or low resolution is a common problem. The authors present a new way to do this faster and with less computing power by teaching a small network to mimic how images get degraded, without needing to retrace through a large image generator. Their approach combines this learned degradation with a powerful image generator, leading to faster and better image repairs for various tasks like sharpening and filling in missing parts.
What this means in practice
- •For photo editing software developers: Integrate faster and memory-efficient image repair features that handle various image degradations in restoration tools.
- •For medical image analysts: Improve speed and flexibility in reconstructing degraded medical images where degradation varies and must be learned for accurate restoration.
Authors
Abduragim Shtanchaev, Arip Asadulaev, Luiza Labazanova, Aidar Alimbayev, Karim Salta, Eric Moulines
Abstract
Latent diffusion models serve as powerful priors for solving inverse problems in image restoration, such as deblurring, inpainting, and super-resolution. Current methods have a trade-off between generality and efficiency. Solvers that are restricted to a fixed set of degradation operators are fast and efficient. Methods that support arbitrary degradation operators are slow and require gradients through the diffusion network. To break this bottleneck, we introduce PASEO (Perturb-And-Solve for Efficient Operator conditioning), a method that uses a small (1M parameters) learned network to degrade diffusion model predictions in latent space. PASEO supports learned degradation operators without back-propagating through the diffusion network. We efficiently sample reconstructions from an approximate posterior by combining the diffusion model's prediction with the observed image. We do this by adding noise and solving linear equations based on a local linear approximation of the learned network, without building or inverting large covariance matrices. Across super-resolution, deblurring, and inpainting on FFHQ and COCO, PASEO achieves strong perceptual quality while running up to 9x faster and using up to 34% less peak memory than the tested baselines, with the same or fewer model evaluations.