Spiking neural networks partition inputs with richer patterns than ReLU nets
Polyhedral Geometry of Time-to-First-Spike Neural Networks
Machine Learning
Summary
This paper explores how spiking neural networks, which use timing of neuron spikes to process information, can represent input-output relationships differently than traditional neural networks. The authors studied a model where each neuron’s firing time is determined by patterns of input spike order, forming complex regions in input space. They mathematically describe these regions as shapes called polyhedra and show that spiking networks can create more varied partitions of inputs than typical ReLU networks. This suggests spiking networks can be more expressive in how they map inputs to outputs.
What this means in practice
- •For machine learning engineers: Design spiking neural networks with enhanced input space partitioning capabilities for event-driven computation scenarios.
- •For neuromorphic hardware developers: Develop hardware architectures that exploit spiking networks' richer partitioning to achieve more expressive temporal processing models.$Commercial implications: This work supports building neuromorphic chips that utilize spike timing for improved computational expressivity, enabling novel AI hardware products.
A theory result. No direct application yet.
Authors
Manjot Singh, Guido Montúfar, Gitta Kutyniok
Abstract
We study the expressivity of spiking neural networks, which provide a natural framework for asynchronous, event-driven computation complementary to conventional feedforward neural networks. We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear, with affine pieces governed by causal feasibility constraints that determine which presynaptic spikes occur before a neuron fires. We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces. We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks. Our theoretical and experimental results show that spiking networks can generate richer partitions of the input space than conventional feedforward ReLU networks.