Efficient neuromorphic learning cuts memory and energy use dramatically
ENCL: Enabling Efficient Neuromorphic Continual Learning with Local Learning Rule and Sub-Network Freezing Strategy
Neural and Evolutionary ComputingArtificial IntelligenceMachine Learning
Summary
Continuously learning new tasks without forgetting old ones is challenging for artificial neural networks, especially when they are running on energy-limited devices. The authors propose ENCL, a technique that helps neural systems learn new information using less memory and energy by training only small parts of the network and using local learning rules instead of complex global updates. This approach avoids storing large amounts of past data and allows faster, low-power learning. Tests on several image datasets show that ENCL matches the accuracy of top methods while dramatically reducing memory, power, and energy demands.
What this means in practice
- •For embedded system engineers: Enable low-power, memory-efficient continual learning in edge devices that update models without retraining full networks or storing large historic data.
- •For ai hardware developers: Design neuromorphic chips with support for local learning rules and dynamic sub-network freezing to reduce training energy and time on real-world tasks.
Authors
Rachmad Vidya Wicaksana Putra, Shuochen Zhou, Hassan Raza, Muhammad Shafique
Abstract
Continual learning (CL) in neural networks is a promising learning approach for updating the knowledge of deployed systems based on new data encountered at run time, that have not been learned during the offline training phase. To enable their power/energy-efficient CL processing, neuromorphic continual learning (NCL) concept with sparse event-based operations have recently been introduced. However, state-of-the-art NCL methods typically employ the memory-hungry data/experience replay, coupled with costly global learning rule (i.e., backpropagation-based surrogate gradient). To address these limitations, we propose ENCL, a novel methodology that enables efficient NCL by employing a lightweight local learning rule and sub-network freezing strategy. ENCL leverages the temporally- and spatially-local learning rule to perform efficient credit assignment for training new tasks, alleviating the expensive cost of global learning rule. ENCL also employs a sub-network freezing strategy based on the given tasks to avoid the needs for network architecture expansion, hence further improving the efficiency and scalability of the NCL training process. To efficiently support this, we train each task with a small number of epochs as well as adjusted learning rate and timestep. Experiment results on task-incremental learning (TIL) scenario show that, our ENCL consistently achieves strong performance across different datasets as compared to state-of-the-art works. Specifically, ENCL achieves competitive accuracy comparable to that of the state-of-the-art data/experience replay, while reducing memory cost by up to 88%, NCL training time by up to 51%, average power consumption by up to 40%, and energy consumption by up to 65% across different benchmarking datasets (i.e., Split MNIST, Split Fashion MNIST, Split CIFAR-10, and Split CIFAR-100).