Papers for

data infrastructure teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Simpler algorithms outperform complex methods for large L1 logistic models

Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets

Abstract: Linear models with an $L_1$-norm penalty remain state-of-the-art for high-dimensional ($d > 1,000,000$) tasks, offering a straightforward method for solving real-world industry problems. Despite their widespread use in industry and utility, many $L_1$ solvers are not effective for general use, are prohibitively slow, and are ineffective in parallelization. This makes them difficult to train in an MLOps pipeline on large industry-scale corpora. In this work, we test several proposed ``state-of-the-art'' solutions from the literature and find that older methods are currently far superior for general use. We also identify several recommendations for academics to perform research that avoids erroneously overconfident results, which can prevent the transition to production use. Equally surprising, we find that a new and simple baseline, using LBFGS on a sub-gradient, is highly effective with minor tweaks, despite being dismissed in the literature for theoretical non-convergence. In practice, we find it is an easier-to-support and easier-to-scale method for production use.

Mon 21 SeptMachine Learning

The gist

Training certain statistical models that use a special kind of penalty, called L1 regularization, can be very slow and hard to run on big datasets. The authors tested various popular methods and found that older, simpler algorithms actually work better for general use. Surprisingly, they show that a straightforward technique called LBFGS applied to a sub-gradient works well in practice, even though theory says it might not converge. This method is easier to scale and maintain in real-world production systems.

Open → 2609.23995v1