Spike-driven model improves robot vision language action with less energy
Spike-driven Vision-Language-Action Model
Computation and Language
Summary
Robots need to understand both what they see and what people say to perform tasks. Existing models that do this usually use big, power-hungry computers. This paper presents a new approach that uses spiking neural networks, which work more like real brains and use much less energy. The authors created a system that processes visual and language inputs efficiently and generates actions for robots with less computing power. Their experiments show it works well while saving energy compared to previous methods.
What this means in practice
- •For robotics engineers: Build energy-efficient robots that interpret visual scenes and language instructions for manipulation tasks using spike-driven neural models.
- •For embedded system developers: Design resource-constrained humanoid or industrial robots with low-latency multimodal perception and control based on spiking neural architectures.
Authors
Shuai Wang, Malu Zhang, Mingquan Liu, Weihui Dai, Dehao Zhang, Jieyuan Zhang, Yimeng Shan, Zijian Zhou, Yang Yang
Abstract
Vision-language-action (VLA) models bridge multimodal understanding and robotic control, advancing the dominant paradigm for embodied intelligence. However, most existing models rely on large Transformers, whose latency and energy costs hinder deployment on resource-constrained platforms. Through sparse event-driven computation, spiking neural networks offer a promising paradigm for high-performance and energy-efficient computing. Here, we propose the first Spike-driven VLA framework enabling end-to-end direct training for robotic manipulation, which mainly comprises three core components. First, we develop spiking visual and instruction encoders for multimodal perception, encoding visual observations and language instructions into sparse, reliable spike representations for subsequent cross-modal fusion. Then, we introduce Multi-Winner Spike Fusion for instruction-guided scene understanding, using bidirectional top-$k$ winner-take-all spike routing to suppress background interference and yield fused memory. Finally, we propose a Spike Action Chunking Transformer that incorporates spiking cross-attention over the fused memory and the current robot state, enabling efficient end-to-end generation of continuous action chunks for robotic control. Extensive experiments on LIBERO and Meta-World demonstrate that Spike-driven VLA achieves competitive performance with fewer parameters and lower estimated inference energy than conventional VLA models. This work establishes a foundational framework for neuromorphic VLA modeling, paving the way for future advances in resource-efficient embodied intelligence.