High fidelity diffusion model improves smooth zoom between dual cameras
ZoomDiff: A High-Fidelity Diffusion Model for Dual-Camera Smooth Zooming
Computer Vision and Pattern Recognition
Summary
Digital zooming between two cameras often looks jumpy or unnatural because the images don’t blend well. The authors found that existing methods struggle to fix this due to big differences between camera views and complex changes in the images. They created ZoomDiff, a new AI model that carefully combines details from both cameras to make much smoother and clearer zoom transitions. Tests showed ZoomDiff makes better-looking zooms than current state-of-the-art methods.
What this means in practice
- •For mobile device manufacturers: Create smooth and natural digital zoom transitions in dual-camera smartphone cameras, improving user video and photography experience.$Commercial implications: Enables smartphone makers to offer higher quality zoom features that blend dual camera images seamlessly, enhancing product appeal.
- •For video editing software developers: Integrate ZoomDiff technology into video editing tools to produce smoother zoom effects when combining footage from multiple cameras.
Authors
Jiayi Zhang, Renlong Wu, Yukang Ding, Sibin Deng, Wangmeng Zuo
Abstract
Digital zoom transitions between dual cameras often exhibit conspicuous discontinuities in geometric structure and chromatic consistency, degrading the user experience. While recent dual-camera smooth zoom (DCSZ) methods attempt to mitigate this by fine-tuning frame interpolation (FI) models on DCSZ data, they struggle with the large cross-view disparities and complex geometric transformations. Considering that the generative prior of diffusion models is suitable for addressing this problem, we explore their application to DCSZ. However, naively applying existing diffusion-based FI models still yields low-fidelity transitions due to insufficient conditional guidance, high-frequency information loss during VAE encoding, as well as inadequate temporal consistency. To address this, we propose ZoomDiff, a high-fidelity diffusion model that leverages dual-camera inputs in both latent and pixel spaces for photo-realistic transitions. Specifically, we first strengthen dual-image conditional guidance during the multi-step denoising process to improve geometric consistency. Then we inject flow-aligned multi-scale features from the VAE encoder into the VAE decoder to recover high-frequency details, where flow-guided temporal consistency supervision are introduced to produce more smooth transitions. Extensive experiments on both synthetic and real-world datasets demonstrate that ZoomDiff outperforms state-of-the-art methods quantitatively and qualitatively. Project page: https://jiayi-hit.github.io/ZoomDiff.github.io/.