Direct feedback alignment reveals common error collapse slows learning
Common-Mode Collapse and Recovery in Direct Feedback Alignment
Machine LearningNeural and Evolutionary Computing
Summary
Training certain neural networks can get stuck because they share a common error pattern that makes hidden units saturate and stop learning effectively. The authors found that this happens when the network’s hidden units respond too similarly, a problem they call 'common-mode collapse.' They show that adjusting the way the output error is fed back and how learning rates are set can reduce this problem and speed up training. Their study tested different settings, networks, and datasets to understand when and why this collapse happens.
What this means in practice
- •For deep learning engineers: Improve training speed of neural networks by calibrating output error feedback to avoid common-mode collapse.
- •For machine learning platform teams: Optimize learning algorithms by incorporating batch mean error subtraction to reduce saturation effects and improve convergence.
Authors
Varun Reddy, Bernardo L. Sabatini, Houman Safaai
Abstract
Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.