Target propagation offers faster training with fewer backward passes
Trust Your Targets: Is Target Propagation Block-Diagonal Gauss-Newton?
Machine Learning
Summary
Training deep neural networks usually relies on backpropagation, which sends error signals backward through all layers. This paper studies target propagation, an alternative method that assigns local goals to each layer based on the overall task. The authors show target propagation can be seen as a form of Gauss-Newton optimization under a special norm, and propose a new way to efficiently solve its subproblems using conjugate gradients. Their method matches the accuracy of common optimizers but requires fewer training steps, making it stable and efficient especially for deep networks.
What this means in practice
- •For machine learning engineers: Train deep neural networks faster and with fewer backward passes by implementing conjugate gradient target propagation.
- •For deep learning framework developers: Incorporate conjugate gradient target propagation as an alternative optimizer to improve training efficiency and stability in multilayer perceptrons.
Authors
Giuseppe Castiglione, Christopher Buckley, Ivor Simpson
Abstract
Target propagation (TP) is a backpropagation alternative in which local targets carry the global learning signal. Prior works claim that TP performs Gauss-Newton (GN) optimisation on these targets, but the correspondence has only been established under unrealistic assumptions where hitting them exactly is trivial. Practical variants instead descend on a least-squares problem via a single gradient step. In this paper, we rederive TP from a trust region under an implicit, previously unexamined norm, and introduce CGTP (Conjugate Gradient Target Propagation), which uses conjugate gradients to solve the layer-local subproblems in parallel. A single adaptive parameter jointly controls this trust region and a matching bound on the weight update; KFAC emerges exactly as the special case where the per-sample Jacobians are replaced by their batch mean, giving its factored damping a variational grounding. On MNIST and Fashion-MNIST MLPs, CGTP matches the accuracy of ADAM and KFAC while needing orders of magnitude fewer outer iterations than ADAM (about 10x fewer backward-pass-equivalents overall), while remaining stable across depth.