Integer-only training cuts neural network memory use by four times
Memory-Efficient Integer-Only Training of Deep Neural Networks
Artificial Intelligence
Summary
Training deep neural networks usually needs lots of memory and floating-point calculations, which can be costly and slow. The authors developed a method that uses only integer numbers (INT8) and integer math for every part of training, shrinking memory use by about four times. Their approach works on popular network types and standard image benchmarks without losing much accuracy. This could help run neural network training more efficiently on devices with limited resources.
What this means in practice
- •For machine learning engineers: Reduce memory requirements for training standard deep networks by switching to fully integer computations with INT8 tensors.
- •For embedded device developers: Implement neural network training on low-resource hardware by using integer-only operations to save memory and computational power.
Authors
Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri
Abstract
Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural networks, in which every tensor (weights, gradients, and activations) is stored as INT8, and every operation (forward pass, backward pass, weight update, optimizer update) is performed in integer arithmetic. Specifically, tensors are stored as signed integer mantissas paired with explicit power-of-two exponents, on which integer-only operations are performed using five custom composable building blocks (products, sums, divide-by-constant, nonlinear functions, and weight updates). This allows complete implementation of trainable integer layers, including convolutions, batch and layer normalization, softmax, and GELU. A per-group velocity mechanism with independent exponent tracking preserves gradient precision across neurons and channels with diverse magnitudes. As every tensor fits in INT8, training memory is roughly 4x smaller than the FP32 baseline. We validate our training algorithm on CNNs (ResNet-18/34/50/101, MobileNetV2) and Vision Transformers (ViT-Ti / B), demonstrating for the first time that training neural networks with INT8 tensors and integer arithmetic can achieve competitive accuracy on CIFAR-10, CIFAR-100 and ImageNet at a reduced memory budget.