Asynchronous method boosts efficiency in split learning for large model training
CommuteProp: Decoupled Training for Communication Bound Split LLM Fine-Tuning
Distributed, Parallel, and Cluster Computing
Summary
Training large language models often involves splitting work between clients and servers, but current methods make clients wait too long, wasting time. The authors propose CommuteProp, which lets clients and servers work at the same time by separating tasks and overlapping communication with computation. This approach speeds up training without losing accuracy and helps keep data private. Their tests show it works well while also improving other privacy tools.
What this means in practice
- •For cloud platform engineers: Increase training speed for distributed fine-tuning of large AI models by overlapping communication and computation.
- •For privacy-focused ai developers: Enhance privacy-preserving model fine-tuning workflows with asynchronous split learning that reduces idle time without accuracy loss.
Authors
CHEN Ding, LUO Haochen, LIU Chen
Abstract
Split learning has emerged as a promising paradigm for privacy-preserving LLM fine-tuning, yet its practical deployment is severely hindered by the sequential communication-computation bottleneck. In conventional synchronous pipelines, clients remain idle while waiting for server-side gradients, resulting in substantial training inefficiency. We propose CommuteProp, an asynchronous split-learning algorithm that decouples the training process into two concurrent phases: a cross-block forward-backward pass and an in-block weight update. By overlapping computation with communication, CommuteProp reduces the marginal cycle time from a sum of all stage latencies to the dominant computational bottleneck. We provide a rigorous asynchronous error and convergence analysis. Moreover, we derive an NS preconditioner method based on our analysis to further mitigate staleness-induced noise. Comprehensive experiments indicate that our algorithm achieves substantial throughput gains while maintaining accuracy comparable to synchronous methods; furthermore, it functions as a plug-and-play module that not only accommodates but actively enhances existing privacy enhancement methods.