Xronos speeds up large language model fine-tuning on diverse edge CPUs
Xronos: Heterogeneity-Aware Tensor Parallelism for Collaborative LLM Fine-Tuning on Edge CPUs
Distributed, Parallel, and Cluster Computing
Summary
Fine-tuning large language models on many edge devices, like smart home hubs or car computers, keeps data private but is slow. The authors found that existing methods designed for GPUs don't work well on CPUs because the CPU has to do both computing and talking at the same time, causing delays. They created Xronos, a new way to divide work that knows some devices are faster than others, so no one waits too long. This approach cuts fine-tuning time and idle waiting while keeping model accuracy.
What this means in practice
- •For embedded system engineers: Improve fine-tuning speed of large models on varied CPU-based edge devices like IoT gateways and smart hubs by reducing idle times.
- •For edge computing platform developers: Build collaborative training frameworks that balance workload on heterogeneous edge CPUs, enabling faster model adaptation with local data.
Authors
Wonmi Choi, Sunjae Park, Dohyeok Kwon, Zhixiong Niu, Yeonho Yoo, Chuck Yoo, Gyeongsik Yang
Abstract
Collaborative fine-tuning on edge devices adapts large language models to domain-specific data while keeping each device's data local. State-of-the-art (SOTA) collaborative fine-tuning techniques are largely designed for GPU-based edge devices and rely on pipeline parallelism (PP). However, many edge platforms, including IoT gateways, smart-home hubs, and in-vehicle computers, are primarily CPU-based. This paper reports that PP is ineffective on CPU-based edge devices because the same CPU handles both model computation and communication, which causes severe CPU contention. Our analysis shows that this leads to 5.75$\times$ higher computation stall ratios than on GPU devices on average. Tensor parallelism (TP) can alleviate this contention by separating computation and communication, but existing TP techniques assume homogeneous devices. On heterogeneous CPU edge devices, we find that this assumption causes faster workers to remain idle for up to 34% while waiting for slower devices at synchronization points. To address the limitations, we propose Xronos, a collaborative fine-tuning framework for heterogeneous CPU edge devices. Xronos uses TP as its execution backbone and combines lightweight profiling with heterogeneity-aware tensor partitioning to reduce the straggler bottleneck. Across diverse devices, models, and benchmark tasks, Xronos reduces fine-tuning time by 18% (TP) to 56% (PP) and the ratio of device idle time by $\sim$5.9$\times$ over SOTA techniques, while maintaining the accuracy.