Recurrent neural networks gain stability and modularity with brain-inspired constraint
Leveraging Dale's Principle as an Inductive Bias in Recurrent Neural Networks
Machine Learning
Summary
Artificial neural networks work differently from real brains because they don’t follow a rule called Dale’s principle, where each neuron is only excitatory or only inhibitory. The authors created a new kind of recurrent neural network that splits neurons into excitatory and inhibitory types, like in the brain. This change helps the networks learn more stably and perform as well or better than usual networks. Their new model also produces connections that are simpler and organized, which could help in building smarter AI.
What this means in practice
- •For ai engineers: Improve training stability and modularity in recurrent neural networks used for multi-task or continual learning scenarios.
- •For robotics developers: Develop reinforcement learning agents with more stable and interpretable internal dynamics by incorporating excitatory-inhibitory neuron separation.
Authors
Jiyi Wang, Chenxiao Yang, Yujia Zhao, Jingzhao Zhang, Lu Mi
Abstract
One of the fundamental differences between biological neural networks and artificial neural networks (ANNs) is that biological neurons obey Dale's principle: each neuron projects either exclusively excitatory or exclusively inhibitory outputs. While this constraint is central to biological circuits, previous attempts to impose Dale's principle in ANNs have often led to training instability or degraded performance. In this work, we introduce EISep RNN, a recurrent neural network architecture that implements Dale's principle through a learnable excitatory-inhibitory separation. Across a series of learning settings, including multi-task learning, continual learning, and reinforcement learning, EISep achieves more stable training dynamics, competitive or improved performance, and sparser, more modular connectivity than vanilla RNNs and previous Dale-constrained models. EISep also exhibits stable and informative latent dynamics. These results establish Dale's principle as a biologically grounded and computationally effective inductive bias for recurrent neural networks.