Foundation models improve hyperspectral image unmixing with resolution fixes
Benchmarking Hyperspectral Foundation Models for Hyperspectral Unmixing
Computer Vision and Pattern Recognition
Summary
Hyperspectral images capture lots of details about materials but often mix signals from overlapping materials, making it hard to tell them apart. The authors studied modern AI models built for these images to see how well they separate mixed materials. They found these models perform very well but lose detail due to how they process images in chunks. By testing simple ways to restore the lost detail, they improved the models’ ability to separate materials accurately.
What this means in practice
- •For remote sensing analysts: Improve material separation in hyperspectral imagery for environmental monitoring using enhanced foundation models with detail restoration.
- •For precision agriculture teams: Enable more accurate crop and soil analysis from hyperspectral data by using foundation models optimized for better feature resolution.
Authors
Edgard Dabier, Christophe Kervazo, Pietro Gori, Florence Tupin
Abstract
Several foundation models dedicated to hyperspectral images have recently been made available. These models are trained on large unlabeled datasets and exhibit strong performance on many hyperspectral imaging tasks, such as classification or denoising. Nonetheless, their performance for hyperspectral unmixing -- the task of separating mixed spectra of overlapping materials in a hyperspectral image -- remain understudied. This might partly be due to the fact that most of them rely on vision transformer backbones, including patchification, leading to a feature resolution problem. While hyperspectral unmixing already arises from the low resolution of hyperspectral images, this patchification step potentially makes the problem even more ill-posed. Therefore, in this work, we aim to answer two questions: 1) \emph{how do foundation models perform in hyperspectral unmixing?}; 2) \emph{how to tackle the feature-level loss of resolution?} To answer the first question, we benchmark foundation models for unmixing, showing that they can reach state-of-the-art performance on four hyperspectral unmixing datasets. To answer the second question, we compare several feature upsampling approaches and empirically show that using a simple one can lead to high performance results. The code is available at https://gitlab.telecom-paris.fr/ring/hfm-hsu.git.