Memory saving method speeds up deep learning training with careful state tracking
State-Aware Activation Recomputation with Replay Contracts and Selective Preservation
Artificial Intelligence
Summary
Training deep learning models requires saving lots of information, which takes a lot of computer memory. The authors show that when you save less by recomputing parts of the training, you have to be careful to keep the computer’s state the same, or else the results can change. They design a system that tracks and restores just the needed parts of the computer’s memory during retraining, choosing the best way to save memory and speed up training. Their method works well on different workloads, especially when the model writes sparsely to large memory areas.
What this means in practice
- •For machine learning engineers: Reduce memory requirements and speed up training of large neural networks by selectively storing or recomputing model states based on workload characteristics.
- •For high-performance computing teams: Optimize GPU memory use during deep learning workloads that involve sparse memory updates to improve training throughput.
Authors
Annan Zhang, Shengjie Song, Wenshuai Yao, Lihao Zhang, Tianmiao
Abstract
Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a different computation. We make this state explicit. A versioned replay contract specifies what replay must restore, and a joint planner chooses, per segment, between activation retention, whole-buffer snapshots and selective-region preservation, charging each option its measured memory and recovery cost. Independent derivative and byte-level checks validate the contract, and a Torch/JAX cohort retains all 5,900 execution records, including matches, rejections and divergences. On frozen sparse-write H100 workloads, each measured over fifty fresh processes, joint planning reduces paired step time by 18.16% against a correctly costed eager/whole planner at a 44 GiB budget (approximate 97.5% ratio interval $[.8172,.8195]$). At 32 GiB, where the eager/whole planner has no feasible static policy, joint planning is 4.98% faster than fixed regions at 29.94 rather than 26.20 GiB. A complete CIFAR-100 dense-queue application runs seven methods for 100 encoder and 25 probe epochs at six training seeds, with byte-identical test scores within each seed. There, joint planning uses 22.41% more allocated memory than whole snapshots, with a process-time geometric-mean ratio of 1.0080 (seed range $[.9845,1.0191]$). Together, the results map where selective preservation pays off: sparse writes with large activations benefit, while in the dense-queue workload whole snapshots use less allocated memory.