Vision test-time training improves hyperspectral image classification accuracy
Spectral-spatial vision test-time training for hyperspectral image classification
Computer Vision and Pattern Recognition
Summary
Hyperspectral images capture detailed colors beyond what the human eye can see, and classifying these images is hard because we usually have very few labels to learn from. The authors developed a method that adapts a vision transformer model during testing to better understand both the color details and spatial layout in these images. Their approach updates the model on each new image to improve accuracy in identifying objects, like crops or materials, in the images. They tested this on four different datasets and showed strong performance, especially on some datasets where this updating step made a big difference.
What this means in practice
- •For remote sensing analysts: Classify land cover types in hyperspectral satellite images more accurately by adapting the model on new images at test time using spectral-spatial context.
- •For agricultural monitoring teams: Improve crop and vegetation classification accuracy in hyperspectral aerial images by leveraging adaptive spectral-spatial feature mixing during model inference.
Authors
Xiang Zhao
Abstract
Hyperspectral image (HSI) classification requires the representation of contiguous spectral signatures and spatial context under limited labeled supervision. Vision test-time training (TTT) offers an alternative to fixed-weight token mixing by performing self-supervised inner-model updates on the input sequence, but its application to center-pixel HSI classification remains insufficiently characterized. We present ViTTT-HSI, a spectral-spatial adaptation of the vision test-time training (ViT3) architecture. A spectral-aware embedding stem extracts ordered-band information using spectral convolution and forms spatial tokens without reducing their spatial resolution. Two ViT3-based TTT encoder blocks perform patch-conditioned token mixing, followed by the fusion of center-pixel and neighborhood-average features for classification. We evaluate the framework on Indian Pines, Pavia University, Salinas, and Houston 2013 using ten random seeds and dataset-specific training fractions of 1% or 5%. The mean overall accuracies are 97.6390%, 98.8753%, 99.3292%, and 98.4283%, respectively. Ablations reveal a substantial contribution from the TTT inner update on Indian Pines and a smaller positive contribution on Pavia University, while the relative benefits of the spectral stem and feature fusion depend on the dataset. The results demonstrate the feasibility of using patch-conditioned vision TTT for spectral--spatial HSI classification and highlight the role of the inner adaptation mechanism.