Spectral token transfer improves frozen vision language action models
SPECTRA-VLA: Base-Fine Hyperspectral Token Transfer at Frozen Vision-Language-Action Interfaces
Computer Vision and Pattern Recognition
Summary
Some objects look very similar in regular color images but can be told apart using hyperspectral imaging, which captures more light details. The paper presents a way to add these extra spectral details into an already trained vision-language-action model without changing its main parts. This method helps the model recognize categories more accurately, using a new technique to inject spectral information into tokens the model understands. The authors show their method works well on unseen images and helps recover specific spectral shapes tied to object categories.
What this means in practice
- •For robotic system developers: Improve object recognition in robots by integrating hyperspectral data into pretrained vision-language-action models without retraining their main components.
- •For remote sensing analysts: Enhance classification accuracy on hyperspectral aerial or satellite images by adding spectral token information to existing model architectures.
Authors
Chih-Chung Hsu
Abstract
Hyperspectral imaging distinguishes RGB-similar objects, but a pretrained vision-language-action model needs a state its frozen decoder can use. SPECTRA-VLA provides a native-width spectral upgrade without retraining the visual encoder or action decoder. A learned continuous residual enriches each target token through its own normalized false-color reference; projection, norm control and validation-calibrated injection regulate the change. Across 86 held-out acquisitions, category balanced accuracy rises from 0.812 to 0.883. With identical residuals and scale, own-token addition improves reuse of an external category-derived probe on frozen decoder logits: 0.701 versus 0.590 with pooled replacement, at similar image-disjoint recognition. Five grouped tests covering the same 86 acquisitions recover within-category spectral shape with $R^2=0.802/0.337$ from continuous and posterior-code states. Category-matched spectral swaps further show that own-token composition approximately doubles the target-specific recovery increment after frozen decoding. Spectral enrichment and receiving-token preservation thus work together at the pretrained interface.