Equilibrium propagation enables training large image generators without backpropagation
PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation
Machine LearningEmerging TechnologiesNeural and Evolutionary Computing
Summary
Training big image generators usually needs a method called backpropagation, which is hard to run on some energy-efficient computers. The authors present a way to train these models without backpropagation by using equilibrium propagation, a process that relaxes a network to steady states to learn. Their approach matches the quality of traditional methods on datasets like CIFAR-10, using only simple operations well suited for analog hardware. This could pave the way for faster, low-power image generation in new computing systems.
What this means in practice
- •For machine learning engineers: Implement energy-based generative image models trained without backpropagation for more efficient neural network training on analog or low-power hardware.
- •For embedded systems developers: Deploy image generation models on neuromorphic or analog hardware that uses relaxation dynamics to replace sequential computation steps, improving power and speed.
Authors
Alex Gower
Abstract
Generating an image with a diffusion or flow-matching model typically involves integrating a velocity field for hundreds of sequential steps. That computation suits a GPU poorly and an analog neuromorphic substrate well, where one fast, low-power physical relaxation could replace the whole sequence. Two things are needed: a generative algorithm whose computation is a relaxation, and a way to train it without backpropagation, whose global gradient transport is hard to realise in analog. Energy Matching supplies the first, sampling by gradient-descent relaxation on a time-independent potential - but obtaining the gradient that descent follows is itself a backward pass, once per integration step. We prove that a predictive coding network with a single linear injection term added to its energy relaxes to a state whose visible layer carries that gradient inherently, with an error that vanishes for small injection strengths. An Energy Matching model trained by backpropagation can therefore be loaded into this network unchanged and sampled by relaxation alone, matching the original model's FID to within 0.04 at CIFAR-10 and ImageNet-32 scale. The network can also be trained without backpropagation: Equilibrium Propagation reproduces backpropagation's parameter updates to a cosine similarity of 0.9998, and training end to end with those updates reaches an FID of 3.64 on unconditional CIFAR-10 against 3.54 for backpropagation. To our knowledge this is the first CIFAR-10 generative model trained with Equilibrium Propagation, and the first at competitive quality by any backpropagation-free rule. This quality does not require group normalisation or attention: an architecture of only convolutions and pointwise nonlinearities, both native to an analog crossbar, reaches an FID of 3.51 when trained by backpropagation.