Cross curvature predicts best block update methods in machine learning
Predicting Block-Coordinate Performance via Cross-Curvature
Machine Learning
Summary
Training machine learning models often involves updating groups of parameters in different ways. The paper’s authors created a mathematical way to compare two common update methods by measuring how these groups influence each other, called cross-curvature. Their formula predicts which method will lower the training error more after several steps. They tested their theory on neural networks, federated learning, and low-rank adaptation, and it accurately picked the better method almost all the time.
What this means in practice
- •For machine learning engineers: Optimize training schedules by predicting the better block update method to reduce loss more efficiently over iterations.
- •For distributed system developers: Improve federated learning system performance by selecting the block update strategy that minimizes training loss based on objective geometry.
Authors
Shengkun Zhu, Jinshan Zeng, Zhiqiang Kou, Yongxin Tong, Yang Liu
Abstract
Simultaneous and sequential block updates are two basic optimization strategies used across machine learning, such as neural-network training, federated learning, and low-rank adaptation. Choosing between them is difficult because their relative advantage depends on both the objective geometry and the number of iterations. We develop a unified theory for comparing Jacobi (JC), Gauss--Seidel (GS), and partially sequential deterministic block-gradient updates. Our analysis expresses the one-step loss difference through cross-block curvature, with an $O(η^3)$ remainder, where $η$ is the learning rate. We derive a signed loss comparison after $K$ iterations with $O(Kη^3)$ error under regularity conditions and $ηK\le T$ for fixed $T$, identifying the better method when the predicted difference exceeds this error. We evaluate these formulas along observed training trajectories across different machine learning settings. Over 500 iterations, our theory correctly identifies the lower-loss method in 98.0\% of iterations for the neural network, 83.4\% for federated learning, and 97.6\% for LoRA. Applying the loss recursion at each step using the measured parameter difference raises these rates to 100.0\%, 93.2\%, and 99.6\%, respectively.