Personalized language model training keeps user data private and adapts preferences
Personalized Federated Direct Preference Optimization via LoRA Adapters
Machine Learning
Summary
Training language models with user preferences is tricky because people have different likes and privacy matters. The authors present a way for many users to train a shared model while keeping personal preferences private by adding a small personal part called a LoRA adapter. This special setup helps the shared model learn generally while letting each user’s part capture their unique tastes. Their tests show this method keeps the training stable and accurate for each user without sharing sensitive data.
What this means in practice
- •For mobile app developers: Train personal language assistants with privacy by keeping user preferences local while improving a shared base model.
- •For enterprise ai teams: Customize company-wide AI tools to diverse employee preferences without risking sensitive data centralization.
Authors
Zhanhong Jiang, Soumik Sarkar
Abstract
Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity $κ^2$ in the FedDPO bound with a residual heterogeneity $κ^2_{\mathrm{eff}}$, which is no larger than $κ^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.