Model improves multispectral image quality by better cross-band structure transfer
S3D-Mosaic: Subgradient-Guided Cross-Band Structure Transfer for Spectral Demosaicing
Computer Vision and Pattern Recognition
Summary
Multispectral cameras capture many colors at once but do so with sparse samples that cause image blurriness or artifacts. The authors developed a new method called S3D-Mosaic that transfers detailed structural information between color bands while ignoring brightness differences. This method uses a special mathematical approach and a lightweight filter to extract sharp edges and patterns across colors. Their approach evolves a model to reconstruct clearer, more accurate multispectral images without needing lots of training data or tuning. It works well on benchmark datasets and different camera setups, even real drones.
What this means in practice
- •For multispectral camera developers: Enhance snapshot multispectral images by reducing aliasing artifacts through cross-band structural information transfer.$Commercial implications: Enables manufacturers to offer higher quality multispectral cameras with better out-of-the-box image reconstruction without extensive per-device calibration.
- •For drone operators: Improve multispectral data quality captured by UAVs for environmental and agricultural monitoring without tuning reconstruction per mission.
Authors
Weiqiang Zhao, Xian Li, Yuzhe Gui, Tianzhu Liu, Yanfeng Gu
Abstract
Multispectral filter array (MSFA) imaging enables snapshot spectral acquisition, but its per-band sparse sampling causes severe spatial aliasing. Although missing spatial structures can be inferred across bands, wavelength-dependent contrast variations make direct cross-band transfer unreliable. Existing model-based methods lack explicit magnitude-invariant mechanisms, and learning-based methods require layout-specific training or per-image tuning, limiting generalization. To address these issues, we propose S3D-Mosaic, a model-based spectral demosaicing method built upon a subgradient-guided cross-band structure transfer framework. We first establish a subgradient-guided regularization framework with a generalized cosine-similarity interpretation, providing a convex mechanism for magnitude-invariant information transfer. Then we design a compact Directional Sparse Equivariant Difference Convolution (DSE-DConv) for band-level extraction of orientation-selective, high-frequency structure features at low computational cost. Instantiating the framework with DSE-DConv on spectrally adjacent bands yields a cross-band self-structure similarity regularizer for spectral demosaicing, which we optimize via a dual-scan ADMM algorithm that decouples the bidirectional dependency while propagating magnitude-robust structural information across bands. With DSE-DConv's two parameters trained once via self-supervision, S3D-Mosaic outperforms state-of-the-art unsupervised methods on CAVE and NTIRE 2022, and generalizes robustly to varying MSFA layouts, pan-mosaic fusion, and real-world UAV data with fixed reconstruction hyperparameters. The source code is available at \url{https://github.com/bestwishes43/S3D-Mosaic}.