Teacher network structure shapes how well students learn functions
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Machine LearningArtificial IntelligenceNeural and Evolutionary Computing
Summary
Teaching one neural network to copy another isn’t always equally easy. The authors show that the internal setup of the teacher network affects how likely a student network is to learn the correct function. They find specific teacher setups that make learning easier or harder by creating different patterns in the learning process. By studying these patterns, they suggest ways to adjust learning rates to improve success. This helps make teaching-student neural network studies more relevant to real-world situations.
What this means in practice
- •For machine learning engineers: Improve neural network training success by adjusting learning rates based on teacher network structure insights.
- •For ai platform developers: Design better training protocols for automated model distillation by understanding how teacher network geometry affects learnability.
Authors
Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea
Abstract
Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.