Summary
When multiple computers learn together without sharing data, they can choose different best ways to predict results. The authors studied how these choices affect the combined model in a simple regression setup. They found a natural way for each computer to pick a prediction method similar to the shared one, helping the group reach a balanced solution. Although this collective solution is different from what a single combined dataset would give, the authors explain why and propose a correction step to fix it. They tested this with simulated and real datasets, confirming their theoretical predictions.
What this means in practice
- •For machine learning engineers: Improve federated regression training by implementing client updates that choose heads close to the shared model, yielding more predictable global models.
- •For data platform teams: Correct global model differences in federated regression by exchanging summary statistics once, enhancing accuracy without more data sharing.
Abstract
In federated averaging, local objectives can admit multiple optimal heads, making the aggregate depend on which heads clients return. We study this ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. We introduce a natural selection rule: each client returns the optimal head closest to the broadcast head. We show that global minimization with a vanishing proximal penalty on the head realizes this rule. When the clients' optimal Gram matrices and the initial shared Gram matrix are positive definite, the shared Gram matrix follows a closed recursion and converges to the unique Bures-Wasserstein barycenter of the clients' optimal Gram matrices. Even with this alignment, the limit generally differs from the centralized optimal Gram matrix. We decompose this gap into three positive-semidefinite terms arising from differences in client target means, covariance heterogeneity, and averaging the aligned heads. A correction based on a one-time exchange of target means and covariances recovers the centralized optimal Gram matrix in one round under exact local optimization and the same selection rule. We verify these results numerically in the UFM and test its predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training. In these experiments, ordinary training approaches the predicted barycenter, while a weak proximal penalty improves endpoint agreement and yields trajectories that closely follow the predicted Gram dynamics. The correction moves the final Gram matrices close to the centralized UFM prediction.