Locality and weight sharing influence convolutional neural network learning speed
The role of locality and weight sharing in convolutional neural networks
Machine Learning
Summary
Convolutional neural networks (CNNs) learn by looking at small parts of images and using the same rules across the whole picture. This paper studies how these two ideas—looking locally and sharing weights—affect the way CNNs learn, especially when pictures have patterns that repeat across space. The authors find that sharing weights helps speed up learning when parts of an image are unrelated, but when regions are similar, this advantage decreases. They also show that matching the size of the local area to the problem helps the network learn with fewer examples.
What this means in practice
- •For machine learning engineers: Design neural networks with appropriate local kernel sizes to reduce training data needs when spatial correlations are low.
- •For computer vision developers: Optimize network architecture choices between convolutional and locally connected layers depending on expected image spatial correlations.
A theory result. No direct application yet.
Authors
Francesco Mori, Melanie Weber
Abstract
The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on $P$ input positions with correlation $c$ between positions. We compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension $D$. For uncorrelated positions ($c = 0$), the two students follow the same learning trajectory, with the convolutional student learning exactly $P$ times faster. We also find that locality affects the sample complexity: a student that matches the kernel size of the teacher recovers it from $\mathcal{O}_{D}(D)$ samples, whereas $\mathcal{O}_{D}(D\log(D))$ samples are needed with a larger kernel. For $c > 0$, spatial correlations can reduce the advantage of weight sharing. The asymptotic error of the convolutional student grows with $c$, whereas that of the locally connected student is independent of $c$ and overall lower when evaluated at the same learning rate. When the learning rate is instead tuned to the training horizon, however, the locally connected student wins at early times and the convolutional student at late times, with a transition that we predict analytically.