Second order optimizer improves spiking neural network training
On the second-order optimization for spiking neural networks
Machine LearningComputer Vision and Pattern Recognition
Summary
Training spiking neural networks is hard because their way of working creates tricky problems for usual methods to fine-tune them. The researchers developed a new training method called SpiKFAX that better understands the underlying patterns of these networks, making training more stable and accurate. They tested it on many types of spiking networks and datasets and found it consistently works better than popular choices. This new method helps unlock the potential of these energy-efficient brain-inspired networks.
What this means in practice
- •For embedded system developers: Improve training stability and accuracy of energy-efficient spiking neural networks in low-power devices by using the SpiKFAX optimizer.
- •For neuromorphic hardware engineers: Enhance the optimization process of spiking neural networks deployed on neuromorphic chips to better capture their discrete and temporal dynamics.
Authors
Ngoc Phu Doan, Ihsen Alouani
Abstract
Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.