Predictive coding method works without activation function derivatives
Analog-Friendly Predictive Coding without Activation Derivatives
Machine LearningArtificial IntelligenceNeural and Evolutionary Computing
Summary
Training neural networks usually requires calculating how outputs change with inputs, which can be hard to do on analog devices. The authors introduce a new way to train predictive coding models that skips this tricky step by using a different mathematical approach tied to the activation function. This makes it easier to run these models on analog hardware without losing their ability to learn well. They tested their method on tasks like classification and generating data, finding it works about as well as traditional methods.
What this means in practice
- •For analog hardware engineers: Design analog neural networks that learn effectively without needing to compute activation function derivatives.
- •For machine learning developers: Build predictive coding models that are easier to optimize on specialized hardware with simpler computation requirements.
Authors
Francesco Innocenti
Abstract
Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.