Simpler algorithms outperform complex methods for large L1 logistic models
Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets
Machine Learning
Summary
Training certain statistical models that use a special kind of penalty, called L1 regularization, can be very slow and hard to run on big datasets. The authors tested various popular methods and found that older, simpler algorithms actually work better for general use. Surprisingly, they show that a straightforward technique called LBFGS applied to a sub-gradient works well in practice, even though theory says it might not converge. This method is easier to scale and maintain in real-world production systems.
What this means in practice
- •For machine learning engineers: Train large L1-penalized logistic models faster and more reliably in industry-scale systems using a simple LBFGS-based approach.
- •For data infrastructure teams: Build scalable pipelines for L1-penalized logistic regression models that parallelize well and are easier to support in production.
Authors
Edward Raff, James Holt
Abstract
Linear models with an $L_1$-norm penalty remain state-of-the-art for high-dimensional ($d > 1,000,000$) tasks, offering a straightforward method for solving real-world industry problems. Despite their widespread use in industry and utility, many $L_1$ solvers are not effective for general use, are prohibitively slow, and are ineffective in parallelization. This makes them difficult to train in an MLOps pipeline on large industry-scale corpora. In this work, we test several proposed ``state-of-the-art'' solutions from the literature and find that older methods are currently far superior for general use. We also identify several recommendations for academics to perform research that avoids erroneously overconfident results, which can prevent the transition to production use. Equally surprising, we find that a new and simple baseline, using LBFGS on a sub-gradient, is highly effective with minor tweaks, despite being dismissed in the literature for theoretical non-convergence. In practice, we find it is an easier-to-support and easier-to-scale method for production use.