Structure-aware adaptation improves light field reconstruction from noisy images
STAR: Structure-aware Test-time Adaptation for diffusion-based light field Reconstruction
Computer Vision and Pattern Recognition
Summary
Rebuilding light field images from a few blurry and noisy pictures is very difficult because the details and angles vary with each scene. The authors propose a method called STAR that adapts a pre-trained model for each new scene by adjusting small parts to better match the scene’s unique structure. This approach improves the reconstruction quality and speeds up processing compared to previous methods. STAR works with different numbers of input images and handles variations in spatial and angular details effectively.
What this means in practice
- •For computational photography developers: Improve the quality and speed of reconstructing light fields from limited noisy focal stack images in camera software.
- •For virtual reality content creators: Enhance realistic rendering of scenes by better reconstructing 3D light fields from sparse input images for immersive experiences.
Authors
Wontae Choi, Ki Ryum Moon, Jae Young Lee, Hyung Sup Yun, Il Yong Chun
Abstract
Light field (LF) reconstruction from limited and noisy focal stack (FS) measurements is a highly ill-posed inverse problem. Although the LF-to-FS imaging geometry is fixed for a given optical setup, LF spatial-angular structure---including within-view spatial details, cross-view angular dependencies, and disparity across views---varies across scenes. Consequently, a fixed pre-trained prior may not optimally capture the spatial-angular structure of each test LF. We propose Structure-aware Test-time Adaptation for diffusion-based light field Reconstruction (STAR), the first test-time adaptation framework for reconstructing an LF from FS. For each test LF, STAR freezes a pre-trained diffusion prior and fits three lightweight adapters to the observed FS to jointly adapt the three components of the LF's spatial-angular structure. STAR outperforms existing state-of-the-art methods in both two- and three-focal-sheet settings, with shorter inference times than those with test-time parameter updates.