Minibatch persistency trades compute for data at large batch sizes
Minibatch persistency, eight years later: what batch reuse costs in steps and joules, and what it saves in data
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Training neural networks often uses a technique called minibatching, where different small sets of data are used for each training step. This paper studies a method called minibatch persistency, where the same set of data is reused for several training steps instead of changing it every time. The authors found that this reuse doesn’t save time or energy but instead reduces the amount of fresh data needed, and this benefit only appears with very large minibatch sizes. For small batches, reusing data can actually require more fresh data, so the method is best when data is limited or expensive to get.
What this means in practice
- •For machine learning engineers: Optimize training setups when data collection costs or limits fresh data availability by reusing minibatches at large batch sizes to save data without losing speed or energy.
- •For cloud service operators: Design pricing or pipelines for paid data usage where minimizing fresh data read is more crucial than compute time or energy savings.
Authors
Matteo Fischetti
Abstract
Minibatch persistency reuses data instead of reading it: rather than drawing a fresh minibatch at every optimizer step, it takes K consecutive steps on the same one. Absorbed into data echoing in 2019, it has carried one objection -- that reuse merely imitates a larger learning rate -- and no baseline tuned as carefully as the method itself. This paper runs the missing test. A pre-registered study trains a 49M-parameter Transformer on FineWeb-Edu at minibatch size B in {32, 128, 512}, 8 seeds per cell, tuning the learning rate separately for every batch size and every arm, against a reuse-free control that changes the sampling and nothing else. Each headline claim is a cost to reach a fixed loss, read on four axes: optimizer steps, fresh tokens, seconds, and joules at the socket. We then replicate on new seeds and a newer GPU generation, and put the three arms on one schedule in steps. The outcome of our study is that what minibatch reuse buys is neither speed nor energy but data, and only at large minibatch size: at B = 32 it reads more fresh tokens than the baseline, not fewer. On steps, seconds and joules it is at best free; and at B = 512, where it looks best, a registered control cannot separate the effect of reuse from the position on the learning-rate schedule at n = 8 seeds. The technique is therefore worth using where fresh data rather than compute is the binding cost: a corpus that runs out, a pipeline that pays per sample, a stream that cannot be rewound. Where the data can simply be read again, spaced epochs do as well or better.