Federated tuning improves personalized model updates for diverse clients
CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning
Artificial IntelligenceDistributed, Parallel, and Cluster Computing
Summary
Tuning AI models on multiple devices without sharing private data is tricky because different users' data vary a lot. The authors found that mixing certain parts of the tuned models together doesn't work well when data differ widely. They created a method called CF-LoRA that separates model parts and groups similar users for better sharing. This method improves performance across various language and vision tasks while keeping communication efficient.
What this means in practice
- •For mobile app developers: Improve personalized model updates on mobile devices without sending user data, enabling better on-device AI performance.
- •For enterprise data teams: Enable efficient collaborative fine-tuning of large models across different company branches while preserving data privacy and handling diverse data.
Authors
Mengjun Yi, Langxing Yang, Suhan Guo, Furao Shen, Jian Zhao
Abstract
Federated LoRA fine-tuning enables parameter-efficient adaptation of pre-trained models without sharing private data, but suffers from two fundamental mismatches under heterogeneous client data: a structural aggregation mismatch caused by independently averaging LoRA factors, and a statistical collaboration mismatch caused by enforcing a single global adapter across divergent clients. To address these issues, we propose CF-LoRA, a clustered federated LoRA fine-tuning framework that combines decoupled factor aggregation with adaptation-aware client clustering. CF-LoRA first learns a globally shared $A$ factor while retaining personalized $B_i$ factors, then identifies clients with similar adaptation patterns based on the cosine similarity of their learned $B_i$ factors, and finally performs intra-cluster $B$-factor aggregation with a frozen $A$ factor. By decoupling LoRA factor aggregation, CF-LoRA preserves the low-rank structure and mitigates the structural aggregation mismatch, while adaptation-aware clustering promotes collaboration among clients with similar adaptation patterns and reduces negative transfer caused by statistical heterogeneity. Experiments on four language tasks and four vision datasets with RoBERTa and ViT show that CF-LoRA achieves the highest average accuracy in both modalities while communicating only one LoRA factor per optimization round.