Summary
Fine-tuning large language models with private data is often hard because it needs a lot of computer memory and power. The authors studied using Apple Silicon computers, which have large shared memory and some new software tools, to see if they work well for this task. They found that improving communication between computers using Thunderbolt links made the fine-tuning faster. When tested on a group of Apple Silicon computers, their improvements made the process up to 1.6 times faster than before. They also compared Apple Silicon with a top Nvidia server and showed Apple Silicon can be a more affordable choice for private fine-tuning.
What this means in practice
- •For enterprise it teams: Use Apple Silicon Mac Studio clusters to fine-tune language models on sensitive data with improved communication optimizations for better speed and cost.
- •For cloud infrastructure engineers: Configure RDMA-over-Thunderbolt links with multi-trunk communication and CPU parallelism for distributed training workflows on Apple Silicon hardware.
Authors
En-Ming Huang, Yao-Ting Hsieh, Hsiang-Yu Tsou, Mu-Chi Chen, Shih-Hao Hung, H. T. Kung
Abstract
Private large language model (LLM) fine-tuning is increasingly important for organizations that need to adapt models using sensitive data, but it often exceeds the memory capacity of commodity datacenter accelerators. Apple Silicon offers a different design point through large unified memory and lower complete-system cost, while recent Apple software support enables distributed execution over RDMA-over-Thunderbolt (TB). This paper studies whether Apple Silicon can serve as a practical platform for private LLM fine-tuning. We characterize RDMA-over-TB communication on Mac Studio nodes, showing that the measured bandwidth is far below nominal TB specifications. Next, we extend Apple's implementation with multi-trunk communication, persistent worker threads, and CPU-side gradient overlap to better exploit multiple direct TB links for LLM fine-tuning workloads. Finally, on a four-node Mac Studio cluster that fine-tunes a Qwen3-9B, our optimizations improve weak-scaling throughput by up to 1.6X over the single-trunk, non-overlapped baseline and reach 936 tokens/s for sequence length 17408. We further compare Apple Silicon with an NVIDIA H100 platform to quantify the trade-off between memory capacity, throughput, and acquisition cost, showing that Apple Silicon can provide a cost-effective solution for private LLM fine-tuning.