Practical optimization methods improve training in machine learning models
Understanding Practical Optimization in Machine Learning
Machine Learning
Summary
Training machine learning models involves many tricky shortcuts and adjustments that classical theories often overlook. This thesis studies how these real-world tricks affect the goals and updates of training algorithms. It shows that some shortcut methods can bias the training but suggests alternatives that guarantee better results. The work also explains how to handle privacy and scaling challenges more reliably in practice. Overall, it helps people understand why certain training tricks work and how to tune them properly.
What this means in practice
- •For machine learning engineers: Improve model training reliability by using unbiased submodel updates and better gradient clipping schedules under heavy noise.
- •For distributed system developers: Develop scalable and privacy-preserving distributed training systems that avoid assumptions of bounded gradients by combining normalization and error compensation.
Authors
Egor Shulgin
Abstract
Modern machine learning relies on practical optimization methods that often differ from the algorithms studied in classical analyses: they train submodels, clip or normalize gradients, use layer-wise non-Euclidean directions, or approximate costly inner computations. This thesis analyzes the objective, assumptions, update, and computation defined by these procedures, rather than those of simplified proxies. For submodel training, we show that Independent Subnetwork Training can converge to a shifted fixed point because sketching before differentiation biases the update for the original objective. Model-Agnostic Sparsified Training instead incorporates the sketch distribution into its objective, which makes sketched gradients unbiased and admits convergence guarantees. For private optimization, we show that adaptive quantile clipping retains bias at any fixed quantile, while suitable quantile schedules recover near-optimal nonconvex rates under heavy-tailed noise. We then combine smoothed normalization with error compensation and obtain distributed private convergence guarantees under standard assumptions, without assuming bounded gradients. For optimizers based on linear minimization oracles, we develop and validate a layer-wise generalized smoothness model and build on it the Gluon framework for Muon- and Scion-style methods. We also show that the approximate orthogonalization in Muon matters: less precise orthogonalization calls for smaller stepsizes, so its precision should be tuned together with the learning rate. Overall, practical methods are better understood when theory retains the features that determine their optimization behavior.