Federated LoRA improves sharing by aligning client input patterns
What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment
Machine Learning
Summary
Training AI models with different groups of devices (clients) working together can be tricky because each group sees different data. Previous methods tried to share parts of the model that looked similar across groups, but this wasn’t reliable when starting from scratch. The authors found that looking at how the input data influences the model’s updates reveals hidden patterns that are more consistent across clients. They created a method called FedSAIL that uses these shared patterns to guide learning, resulting in better performance and less data sent between devices.
What this means in practice
- •For mobile app developers: Improve model updates shared among devices with different local data, reducing communication needs while improving app personalization.
- •For edge ai engineers: Enable more effective federated training by aligning shared model components based on input patterns instead of raw parameter similarity.
Authors
Junye Du, Shuaida He, Long Feng
Abstract
Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.