Privacy enhancing concurrent deduplication improves federated learning reliability
Deduplication-while-Training: A Resilient Paradigm for Privacy-Preserving Cross-Client Deduplication in Federated Learning
Cryptography and SecurityDistributed, Parallel, and Cluster Computing
Summary
Duplicated data across users slows down learning and risks privacy when training big language models on distributed devices. The authors introduce a new method that removes repeated data while the training happens, rather than before. This way, the learning can continue smoothly even if users disconnect or new ones join, making the process more efficient and resilient. Their approach speeds up recovery from faults and supports dynamic participation better than previous methods.
What this means in practice
- •For cloud service providers: Implement federated learning with efficient deduplication that handles user disconnections and new clients dynamically to improve privacy and training speed.
- •For mobile app developers: Build apps that contribute training data securely with ongoing duplicate data removal, allowing seamless learning updates despite unstable network conditions.
Authors
Rongxi Wang, Guanxiong Ha, Chunfu Jia, Yongsheng Lin, Minfen Gao, Hanmiaomiao Wang
Abstract
Cross-client duplicate data in large language model training corpora degrades the efficiency of federated learning (FL) while exacerbating model memorization and privacy risks. Privacy-preserving cross-client deduplication effectively mitigates this issue by eliminating duplicate training data. However, existing schemes all follow a "Deduplication-before-Training" paradigm. This serially coupled paradigm incurs high fault-tolerance costs and lacks support for dynamic client joining. To this end, we propose an unexplored paradigm called "Deduplication-while-Training (DwT)", which enables concurrent deduplication and training. DwT transforms cross-client deduplication from a one-time, globally synchronous preprocessing operation into a continuous online service with state management, concurrent claiming, and failure recovery. By enabling state synchronization and task takeover, it minimizes the impact of client disconnections on the overall training progress while supporting the dynamic joining of clients. We design DwT-FL, a privacy-preserving deduplication system, to support DwT. By designing a concurrent state-claim mechanism and a hot-cold dual-queue scheduling strategy, DwT-FL enables the parallel execution of secure deduplication and model training, while effectively handling client disconnections and dynamic joins. Experimental evaluations demonstrate that, compared to the state-of-the-art scheme, DwT-FL significantly reduces the time overhead of failure recovery and dynamic joining by up to 93.04% and 94.18%, respectively. This provides an efficient and elastic concurrent deduplication scheme for dynamic and unstable FL environments.