SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
2026-08-03 • Neural and Evolutionary Computing
Neural and Evolutionary ComputingMultimedia
AI summaryⓘ
The authors present the SMM Transformer, a special kind of neural network that mimics brain spikes to process multiple types of data like images and text. They designed new parts like PLMP neurons and SMSA attention to make training deep spiking networks more stable and efficient. Their model mixes information from different sources in a smart way using SMoE. Tests show it performs about as well as regular neural networks but uses much less energy in key parts.
Spiking Neural NetworksTransformerPLMP neuronP-STBP algorithmSMSA attentionSpiking mixture-of-expertsMultimodal learningEnergy efficiencySparse activationsSoftmax attention
Authors
Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu, Yu Zhao, Hongzhi Wang
Abstract
Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.