Spiking neural networks reduce communication and improve accuracy with predictive coding
Predictive Suppression Layers for Communication-Efficient Spiking Neural Networks
Neural and Evolutionary ComputingMachine Learning
Summary
Spiking neural networks send signals called spikes to process information, but they often send too many redundant spikes that waste resources. The authors propose a way for these networks to predict which spikes are important to send and suppress the rest, much like only sharing surprising news. This method cuts the communication needed by about three times while actually making the network better at its tasks. This approach helps in settings where communicating data is costly or slow, like in low-power devices or networks with many processing units.
What this means in practice
- •For neuromorphic hardware designers: Reduce communication overhead between processing units in neuromorphic chips by selectively transmitting only unpredictable spiking activity.
- •For iot device developers: Enable wireless IoT devices using spiking neural networks to transmit less data without losing accuracy, saving power and bandwidth.
Authors
Aidin Attar, Michele Rossi
Abstract
Feedforward Spiking Neural Networks (SNNs) typically propagate every generated spike indiscriminately, disregarding whether the information is redundant from an information-theoretic perspective. This lack of selectivity induces high redundancy in inter-layer communication, creating an expensive overhead, e.g., in scenarios involving many-core neuromorphic hardware or communication-dominated Internet-of-Things (IoT) where features are transmitted wirelessly. To address this challenge, we trade localized processing for leaner network channels by introducing a minimal predictive coding framework for SNNs. We propose two layer variants sharing a predictor block: error units, which transmit signed spiking residuals, and predictive suppression, which uses residual magnitude to dynamically gate and forward only unpredictable, "surprising" activity. Evaluated on the N-MNIST and Spiking Heidelberg Digits (SHD) datasets using diagnostic metrics that decouple local processing from cross-layer communication, our new predictive coding layers achieve significant communication savings. Numerical results reveal a three-fold reduction in communicated activity, while increasing the task accuracy for both datasets. The latter finding is notable, and suggests that predictive coding layers not only minimize communication overhead, but also produce output feature vectors with a higher representation power.