Spiking transformers improve token communication with gated propagation
Rethinking Pairwise Token Interaction in Spiking Transformers
Neural and Evolutionary Computing
Summary
Traditional transformers use a method where tokens communicate based on matching pairs of spikes, which can be very sparse and depend on exact timing. The authors show that this method limits how information spreads between tokens in spiking transformers. They propose a new method called Gated Spike Axial Propagation (GSAP) that sends information along horizontal and vertical paths first, then selectively decides what to keep at each token. This change allows for better long-range communication while keeping the system efficient and event-driven.
What this means in practice
- •For neuromorphic hardware engineers: Design spiking transformer models that enable more efficient long-range token communication without dense pairwise spike matching.
- •For real-time signal processing teams: Implement spiking neural networks with improved sparse communication patterns for faster and more scalable event-driven processing.
Authors
Sicheng Shen, Dongcheng Zhao, Zhiyuan Li, Jinyan Yu, Qian Zhang, Dengpeng Xing, Zhitong Zhang, Tielin Zhang
Abstract
Spiking Transformers inherit token interaction mechanisms from conventional Transformers, yet their sparse binary representations fundamentally alter how token-to-token communication is established. In particular, spike-based query-key matching produces highly sparse and input-dependent interaction patterns, coupling information propagation to the instantaneous availability of matching spike events. This motivates a different interaction paradigm in which long-range communication does not rely solely on pairwise spike coincidence. We therefore propose Gated Spike Axial Propagation (GSAP), a spike-native token interaction mechanism that decouples information propagation from context selection. Instead of directly determining communication through query-key matching, GSAP first propagates spike-based context along the horizontal and vertical axes, allowing information to reach distant tokens through structured sequential propagation. A receiver-conditioned gate then determines how much of the propagated context is incorporated at each token, while a lightweight local pathway preserves fine-grained neighborhood information. In this way, GSAP reformulates token interaction as a propagate-then-select process, enabling structured long-range communication while retaining the sparse event-driven nature of spiking representations. Code is available at https://github.com/Fancyssc/GSAP.