Federated learning boosts language model training without sharing data
Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization
Artificial Intelligence
Summary
Training large AI language models often needs lots of shared data, which can break privacy rules. The authors propose Fed-GRPO, a way for many users to help train these models together without sharing their private data. Their method uses the feedback reward from training to decide how to combine learning from different users more efficiently. Experiments show their approach comes close to training with all data in one place while greatly reducing communication needs.
What this means in practice
- •For ai platform engineers: Train large language models across multiple devices without centralizing sensitive data, using Fed-GRPO to improve efficiency and privacy.
- •For communication system designers: Design networking protocols that allocate bandwidth adaptively during federated training to reduce communication load based on reward-driven signals.
Authors
Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng, Mengting Liu, Liangqiong Qu
Abstract
Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) \emph{signal-weighted aggregation} that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) \emph{global reward calibration} that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) \emph{adaptive sparse communication} that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by $32\times$ and supporting up to $621\times$ compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO.