Summary
Choosing how the learning rate changes during training is important for teaching deep neural networks. The authors propose a schedule inspired by the smooth curve of fastest descent in physics, called BrachistoneLR, which tweaks an existing method to better reach its lowest learning rate just at the end of training. They tested this and five other schedules across many models and datasets while keeping everything else constant. Their findings show that smoothly reducing the learning rate helps more than sudden drops or constant rates, especially for harder tasks, and that small differences between the best smooth methods are hard to spot with limited tests. They also find that focusing on the overall shape of the learning rate change is more important than fine-tuning its settings.
learning rate scheduledeep neural networkscosine annealingbrachistochroneimage classificationtraining epochsoptimizerresidual networksmodel evaluationsmooth decay
Abstract
The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.