LeMA scales Levenberg-Marquardt training for billion parameter models
Scalable Levenberg-Marquardt Training Framework for Deep Learning
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Training big neural networks can be slow and require a lot of computer memory, especially with some older powerful math methods. The paper shows how to make one such method, called Levenberg-Marquardt, work efficiently even on very large networks using multiple GPUs. The authors designed a way to split and handle data so computers don’t get overwhelmed by memory needs. This speeds up training significantly while still keeping accuracy, making it possible to use this method on huge neural networks.
What this means in practice
- •For deep learning engineers: Train billion-parameter neural networks faster and with less memory by using LeMA on multi-GPU systems, enabling exact optimization steps without approximations.
- •For high performance computing teams: Deploy scalable distributed algorithms for nonlinear least-squares problems in large-scale machine learning workloads across multiple GPUs with efficient memory use.
Authors
Xianghao Wang, Johan Barthélemy, John Taylor
Abstract
The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. These costs are primarily from the Jacobian matrix, whose dimensions scale with both the number of samples and parameters. This bottleneck prevents its use in large-scale neural networks. Recent efforts scale LMA using preconditioned conjugate gradient (PCG) solvers to avoid materializing the Jacobian. However, PCG only approximates the solution, requiring many network propagations and a task-specific preconditioner. Other studies optimize LMA to reduce costs, but they restrict their methods to specific tasks or network architectures. As a result, LMA remains impractical for training general large-scale neural networks. In this paper, we propose LeMA, a distributed LMA algorithm for large-scale neural network training on multi-GPU platforms. Our contributions are twofold: First, an exact dual formulation of LMA removes the memory dependence on the parameter dimension. Second, a Jacobian sharding strategy computes Jacobian shards on the fly and in parallel to avoid materializing the full Jacobian. For large-scale neural networks, LeMA reduces the memory footprint of LMA from quadratic in the number of parameters to quadratic in the batch size. This makes exact LMA steps feasible for billion-parameter models without approximate PCG solves. Furthermore, for a convolutional neural network with 1,000,022,632 parameters on an 8-GPU node, LeMA achieves a 7.97x speedup over a single GPU, corresponding to a 99.7% strong scaling efficiency.