Adaptive phase-switching cuts communication in federated model tuning
Adaptive Phase-Switching for Communication-Efficient Federated LoRA Fine-Tuning
Machine LearningArtificial IntelligenceDistributed, Parallel, and Cluster Computing
Summary
Training large language models on many devices can send a lot of data back and forth, which slows things down. The authors studied a way to switch how devices send updates based on how much the model is improving, instead of using fixed rules. Their new method, called ReverseAdaptive, reduces data sent by about 40% while keeping the model’s performance nearly the same. It also works well for different sizes of models and across datasets without extra tuning.
What this means in practice
- •For machine learning engineers: Reduce communication costs when fine-tuning large models across multiple devices by adapting update aggregation based on training progress.
- •For cloud infrastructure teams: Optimize network usage for distributed model training by implementing adaptive protocols that save upload and download bandwidth during federated LoRA fine-tuning.
Authors
Jerry Adams Franklin
Abstract
Federated fine-tuning of large language models with low-rank adaptation reduces per-client trainable parameters, but client-to-server communication remains the dominant cost. Existing accounting for federated LoRA protocols omits the asymmetric transition round when a protocol changes aggregation mode, and reports savings that ignore grouped-query attention shapes. This paper measures per-round upload and download bytes for a bidirectional B-only federated LoRA protocol and places five methods, three from prior work, on a single communication-quality frontier. The frontier has a knee, which an adaptive phase-switching aggregator, ReverseAdaptive, locates by monitoring the relative improvement in global training loss against a dimensionless threshold rather than by fixing a phase boundary in advance. On TinyLlama-1.1B-Chat with Alpaca, ReverseAdaptive attains 40.5 percent measured round-trip savings over FLoRA at a held-out instruction-following loss cost of 0.0063. It outperforms FFA-LoRA, which freezes the first of the two LoRA factors at initialization, by 0.0182 in held-out loss, more than twenty times the largest per-method seed standard deviation on that metric, so learning that factor before freezing it produces better adapters. The same threshold transfers across model scales without retuning, and the quality cost of the transition is stable across the two datasets tested.