Multimodal prompts improve tracking in hyperspectral videos
Latent Spectral Prompting for Multimodal Hyperspectral Object Tracking
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Tracking objects in videos can be harder when colors look the same or lighting is poor, but hyperspectral videos capture many different wavelengths of light, giving more detailed information. This paper presents a new method that uses knowledge from tracking models trained on normal color videos to help track objects in hyperspectral videos. It works by creating a compact representation of the spectral information from the whole scene and combining it with a pretrained tracker that understands movement over time. The method improves tracking accuracy on challenging datasets without needing many labeled examples.
What this means in practice
- •For surveillance system engineers: Enhance object tracking accuracy under difficult lighting or camouflage by integrating hyperspectral video analysis with existing RGB tracking models.
- •For remote sensing teams: Improve tracking of environmental phenomena or objects by leveraging spectral information without extensive labeled hyperspectral data.
Authors
Wangshu Cai, Jun Zhou, Lei Wang, Mohammad Aminul Islam, Xu Han, Kuldip Paliwal
Abstract
Hyperspectral videos provide rich spectral signatures for robust tracking under challenging conditions such as camouflage and low illumination. However, limited hyperspectral annotations and the large RGB-hyperspectral domain gap hinder the direct use of modern temporally pretrained RGB trackers. We propose SProTrack, a multimodal framework that transfers temporal knowledge from pretrained RGB trackers to the hyperspectral domain through latent spectral prompting. The key idea is to decouple temporal modeling from spectral adaptation, allowing temporal dynamics to be inherited from RGB foundation models while hyperspectral information is learned from unlabeled data in a self-supervised manner. Unlike band-selection methods that discard most spectral information or prompt-based approaches that rely on local-search crops, SProTrack learns a global, scene-level spectral representation from the full hyperspectral frame using an autoencoder. This latent representation is used to guide spectral prompt extraction while a pretrained RGB tracker preserves strong temporal modeling on RGB-compatible inputs. By encoding spectral-spatial structure into compact latent prompts, SProTrack bridges the RGB-hyperspectral modality gap without extensive retraining or dense supervision. Experiments on HOT2020 and HOT2023 demonstrate that SProTrack achieves the best overall performance, with strong results across all spectral domains.