GPU powered tools speed up large scale math optimizations with random methods
GPU-Enabled Large-Scale Optimization Using Randomized Linear Algebra
Machine Learning
Summary
Solving big math problems quickly on computers can be hard, especially when using many variables or data points. The authors created a software package called rlaopt that makes these large problems faster to solve by combining random math techniques with the power of GPUs, which are special computer chips good at handling lots of calculations at once. Their tool also lets users write math problems in a simple way and supports advanced features like optimizing problem settings automatically. Tests show their approach can speed up tasks such as regression and classification problems. This software is free and open for anyone to use or build upon.
randomized numerical linear algebraGPU accelerationconjugate gradient methodoperator splittingstochastic gradient methodconvex optimizationempirical risk minimizationpreconditioningridge regressionmodeling language
Authors
Pratik Rathore, Zachary Frangella, Parth Nobel, Xuning Hu, Madeleine Udell
Abstract
This paper introduces rlaopt, a PyTorch-based package for large-scale optimization and scientific computing using randomized numerical linear algebra (RandNLA). Despite substantial progress in RandNLA-based algorithms, few implementations combine GPU acceleration with a simple interface for specifying optimization problems. rlaopt addresses this gap by providing GPU-enabled solvers for positive-definite linear systems and convex empirical risk minimization with constraints and regularizers. These solvers use RandNLA to accelerate conjugate gradient (NystromPCG), operator splitting (NysADMM), and stochastic gradient methods (SAPPHIRE). Moreover, rlaopt includes a modeling language that lets users specify problems using natural mathematical syntax. rlaopt automatically checks compatibility with the selected solver and performs the required problem decomposition. The solvers also support differentiation through their iterations, enabling applications such as hyperparameter tuning. Experiments on ridge regression, bounded multinomial logistic regression, and bounded elastic net identify when randomized preconditioning improves performance and demonstrate substantial speedups from GPU execution. The package is open-source under an Apache license, with source code at https://github.com/udellgroup/rlaopt and version 0.1.0 available on PyPI.