Method finds simple local learning rules to replace backpropagation

A method to automatically discover symbolic local learning rules

Machine LearningNeural and Evolutionary Computing

Summary

Training deep learning models usually requires a technique called backpropagation, which is complex and can be energy-intensive. This paper shows a way to automatically discover simpler, local rules that each layer in a network can use to learn without needing full backpropagation. The authors demonstrate their method on small networks and various tasks, showing it can effectively reduce overall errors. Their approach also produces simpler rules that are easier to understand compared to some other methods.

What this means in practice

  • For deep learning engineers: Develop local learning algorithms that reduce energy use and complexity in training multilayer neural networks.
  • For neuromorphic hardware designers: Create hardware implementations of simpler, interpretable learning rules inspired by biological plausibility for efficient on-chip learning.

Authors

Andrea Perin, Fabio Anselmi, Stephane Deny

Abstract

We present a general method for discovering symbolic local learning rules that can replace backpropagation, applicable, in principle, to any deep learning architecture and task. The method automatically discovers local loss functions specific to each layer by (1) constructing a dictionary of monomials of local tensor variables, inspired by Penrose diagram notation, (2) regressing over coefficients weighting these monomials to build layerwise polynomial losses whose gradients together minimize a target global loss. We test the method on classification, reconstruction, and self-supervised learning tasks with small multilayer networks and minimal datasets. Across all tasks, the discovered rules induce non-trivial interactions between layers, which lead to significant reductions in the corresponding global losses. Furthermore, the method selects sparse combinations of monomials, finding local rules that are more interpretable than those generated by MLP-based alternatives. The potential implications of our method range from energy-efficient deep learning to the discovery of biologically plausible learning mechanisms.