Wider networks enable simple layer-wise training to match full backpropagation
Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning
Machine LearningArtificial IntelligenceComputer Vision and Pattern RecognitionNeural and Evolutionary Computing
Summary
Training deep neural networks usually involves adjusting all layers together, which can be complex. The authors studied a simpler training method that updates each layer one at a time without full coordination. They found that when networks are made wider (with more units per layer), this simpler method can perform just as well as the usual complex way. They also discovered that wider networks trained this way create better internal features for understanding data. This suggests making networks wider can help simpler training methods compete with traditional ones.
What this means in practice
- •For machine learning engineers: Train wide convolutional networks efficiently using greedy layer-wise updates rather than full end-to-end backpropagation without losing performance.
- •For ai hardware designers: Simplify training hardware by supporting local layer updates for wide networks, reducing complexity from full network error propagation.
Authors
Syon Mansur, Joel Zylberberg
Abstract
End-to-end backpropagation has been the dominant mode of training in deep learning, allowing for the coordination of parameter updates across layers of a neural network. Prior studies have explored alternative -- and, in some cases, simpler -- training mechanisms, showing that they can sometimes achieve performance similar to backpropagation. However, the architectural conditions under which locally optimized networks, which avoid end-to-end backpropagation of error, can learn representations comparable to those learned through end-to-end training remain unclear. We aim to answer this question in the context of self-supervised learning, an important framework for large-scale pretraining in artificial intelligence. Here, we investigate how network width and depth affect the efficacy of greedy layer-wise and end-to-end self-supervised training in convolutional networks. We find that in wider networks, the benefits of end-to-end backpropagation over greedy layer-wise training shrink: in relatively shallow and very wide networks, we even observed higher performance in models trained with greedy layer-wise training. Subsequent analysis of the representations formed by these networks shows that very wide greedy-trained networks exhibit more favorable representational geometry than do networks trained end-to-end with backpropagation. This work shows that width can compensate for restricted credit assignment and identifies differences in representational geometry as a potential mechanism for their improved performance.