Backward coupling improves cross-layer learning in deep neural networks
Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning
Machine Learning
Summary
Deep neural networks learn by passing information forward and adjusting connections backward during training. The paper studies how feedback from later layers can reshape earlier layers’ features, something absent in older layer-wise methods. The authors develop a theory and algorithm that captures this backward influence through spectral (frequency-like) modes and test it on image recognition tasks. Their method improves feature learning and prediction accuracy compared to previous forward-only approaches.
What this means in practice
- •For deep learning engineers: Incorporate backward coupling dynamics to improve feature training in multilayer neural networks and increase classification accuracy on image tasks.
- •For computer vision teams: Use improved spectral feature learning algorithms to enhance model performance on binary image classification benchmarks like CIFAR-10.
Authors
Yatin Dandi, Matteo Vilucchio, Hugo Tabanelli, Florent Krzakala
Abstract
During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.