Spiking transformers get better high-frequency focus with contrastive attention
Contrastive Attention Mitigates Spectral Bias in Spiking Transformers
Artificial Intelligence
Summary
Spiking transformers are a type of energy-saving computer model, but they struggle to handle quick changes in data. The authors found that parts of these models act like filters that lose detailed information. To fix this, they created a new module inspired by how the human eye detects edges, which helps the model keep important detailed signals. Their module improves the performance of spiking transformers on tasks like recognizing images, understanding scenes, and tracking events, while also being more efficient.
What this means in practice
- •For embedded system developers: Integrate the contrastive attention module to improve energy-efficient visual recognition in low-power devices without losing detail in complex images.
- •For robotics engineers: Enhance event-based tracking in robots by incorporating the module to maintain fine details while preserving energy efficiency.
Authors
Xiaoli Liu, Malu Zhang, Yang Yang
Abstract
Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.