Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
2026-08-10 • Machine Learning
Machine LearningArtificial IntelligenceDistributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors study a way to efficiently update large language models in federated learning by breaking updates into two small matrices, A and B. They explore whether it’s better to share matrix A with all clients while keeping B local, or vice versa, and find that each choice affects how well the model fits data across clients. They propose a method called FedAS-LoRA that chooses the best sharing strategy before training by using a new metric called RSS, which predicts how suitable the shared parts are. Their experiments show that this approach improves fine-tuning performance on various tasks and setups.
Low-rank adaptation (LoRA)Large language models (LLM)Federated learningMatrix factorizationFine-tuningClient-specific updatesProjection residualRank-Aware Shared-Subspace Sufficiency (RSS)Subspace sharingFedAS-LoRA
Authors
Xinyi Xu, Bingnan Xiao, Shuang Qin, Gang Feng, Tony Q. S. Quek
Abstract
Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.