Dynamical batching improves neural network training by cycling class focus
Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks
Machine Learning
Summary
Training neural networks usually involves mixing all types of examples evenly in each batch, but this paper shows that changing the focus to one class at a time in cycles can help the network learn better. This method, called Dynamical Batching, helps networks that otherwise struggle to learn complex data patterns by letting them switch more efficiently between class-specific states. The researchers found that this cycling changes how the network's internal parts behave, making some connections more persistent. They tested this idea on simple synthetic data and more complex image recognition tasks, with modest improvements in accuracy.
What this means in practice
- •For machine learning engineers: Improve training of capacity-limited neural networks by cycling class proportions in batches to achieve better data fitting and accuracy.
- •For computer vision developers: Enhance image recognition model performance on CIFAR-10 by training with class-focused batch cycling without changing optimizer or loss functions.
Authors
Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-García
Abstract
We investigate whether cyclically varying the class composition of training batches can modify both the performance and the internal organization of neural networks. Motivated by nonequilibrium systems, in which cycling between related sub-goals can make switching between their solutions increasingly efficient, we introduce Dynamical Batching (DB), a sampling protocol that periodically emphasizes each class while leaving the loss and the optimizer unchanged, and compare it with Uniform Batching (UB), in which every mini-batch has equal class proportions. In the regime where standard training fails, DB changes the outcome: on a three-arm spiral, a capacity-limited MLP that plateaus far below perfect accuracy under UB fits the training data under DB. Tracking individual weights, we find that under DB the change between class-specialized states is carried by a smaller set of weights than under UB, and that this set is markedly more persistent from cycle to cycle, on every synthetic dataset we test and in a wider network that fits the data under both protocols. This organization is a signature of the protocol rather than a predictor of its benefit: it appears equally where DB leaves accuracy unchanged. On CIFAR-10, the same protocol modestly improves the test accuracy of a channel-narrowed ResNet-10, a standard residual architecture, and of a second capacity-limited convolutional network. These results show that how class emphasis is distributed in time can shape both the solution a network reaches and the internal structure of the network.