Gpu accelerated package speeds up large scale optimization
GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra
Machine Learning
Summary
Solving big math problems on computers can be very slow, especially when working with large data. The authors created a new software tool called rlaopt that speeds up these calculations by using smart math shortcuts called randomized linear algebra and running them on powerful graphics cards (GPUs). Their tool makes it easier to set up and solve optimization problems, like finding the best fit for data, and can even help tune settings automatically. They showed that their approach is much faster for some common problem types and made the software freely available.
What this means in practice
- •For machine learning engineers: Speed up training of large-scale regression and classification models using GPU-accelerated randomized methods.
- •For data scientists: Rapidly solve linear systems and constrained optimization problems with easy-to-specify models in PyTorch.
Authors
Pratik Rathore, Zachary Frangella, Parth Nobel, Xuning Hu, Madeleine Udell
Abstract
This paper introduces rlaopt, a PyTorch-based package for large-scale optimization and scientific computing using randomized numerical linear algebra (RandNLA). Despite substantial progress in RandNLA-based algorithms, few implementations combine GPU acceleration with a simple interface for specifying optimization problems. rlaopt addresses this gap by providing GPU-enabled solvers for positive-definite linear systems and convex empirical risk minimization with constraints and regularizers. These solvers use RandNLA to accelerate conjugate gradient (NystromPCG), operator splitting (NysADMM), and stochastic gradient methods (SAPPHIRE). Moreover, rlaopt includes a modeling language that lets users specify problems using natural mathematical syntax. rlaopt automatically checks compatibility with the selected solver and performs the required problem decomposition. The solvers also support differentiation through their iterations, enabling applications such as hyperparameter tuning. Experiments on ridge regression, bounded multinomial logistic regression, and bounded elastic net identify when randomized preconditioning improves performance and demonstrate substantial speedups from GPU execution. The package is open-source under an Apache license, with source code at https://github.com/udellgroup/rlaopt and version 0.1.0 available on PyPI.