Local learning methods face limits compared to backpropagation in deep networks
The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Machine Learning
Summary
Backpropagation is a common way to train deep learning models, but it requires information flowing backward through the network. The Forward-Forward Algorithm tries to avoid this by training layers separately using only local information, which simplifies training. The authors found that this method struggles because layers affect each other unpredictably and their internal representations lose diversity as the network gets deeper. This limits how well the model can learn compared to standard backpropagation, especially in deeper networks.
What this means in practice
- •For deep learning engineers: Identify limitations in local training methods when designing deep neural network training schemes that avoid backpropagation.
- •For hardware accelerator designers: Understand fundamental bottlenecks in local learning algorithms to better evaluate hardware support for alternative training methods.
A theory result. No direct application yet.
Authors
Zhaoxian Wu, Haichuan Liu, Tianyi Chen
Abstract
The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.