Summary
Spiking neural networks (SNNs) are a type of brain-inspired technology that can work with less energy by using bursts of activity instead of continuous data. Designing these networks is complicated because their components interact in complex ways. The authors use large language models to help automatically create new SNN designs by evolving computer programs, aiming to find architectures better suited to the unique features of spike-based communication. Their search found new SNN structures that outperform some existing methods on text tasks while using much less energy. The new designs and code will be shared publicly for others to use.
What this means in practice
- •For machine learning engineers: Develop energy-efficient spiking neural network models for sequence tasks improving performance and reducing computation costs in AI systems.
- •For embedded systems developers: Integrate newly discovered SNN architectures to enable low-power processing of sequential data on devices with limited energy resources.
Authors
Ruoyu Zhao, Jiaqi Wu, Chenyu Zhu, Zhichao Lu
Abstract
Spiking neural networks (SNNs) offer low-energy sequence modeling through sparse, event-driven computation. However, interactions among spike encoding, neuronal dynamics, and information propagation complicate architecture design. Existing SNN sequence models often adapt artificial neural network (ANN) architectures designed for real-valued activations, potentially underusing spike-based communication and temporal state updates, motivating automated discovery of native SNN architectures. Most evolutionary neural architecture search (ENAS) methods operate within predefined configuration spaces, limiting discovery to mechanisms expressible within those spaces. We introduce OpenArchEvo, which uses large language models (LLMs) to evolve executable architecture code in an open program space under spiking-projection constraints. In this space, code differences need not reflect architectural novelty, while direct performance evaluation requires costly training. We construct a three-view representation spanning code, design rationale, and a behavioral fingerprint to support novelty estimation and performance prediction. The search treats predicted performance and estimated novelty as two objectives, using surrogate predictions to select candidates for expensive training evaluations. With an estimated candidate-training cost of 132 V100 GPU-days, the search uncovers multiple native SNN architectures, exemplified by three designs featuring mechanisms such as spike-activity-dependent control of state updates and residual pathways. The discovered NeuroGate surpasses the ANN DeltaNet on WikiText-103, and the discovered architectures reduce estimated architecture-level arithmetic energy by up to 50.6x (LoopMem) relative to a common dense Transformer (ANN) baseline. All code and all discovered architectures will be made publicly available soon.