Pre-training large language models across multiple supercomputers worldwide
Cross-Facility LLM Pre-training on HPC: Elastic Aggregation, Data Leasing, and Queue-Aware Placement
Distributed, Parallel, and Cluster Computing
Summary
Training large language models usually happens on one set of computers at a single place. This paper shows how to combine computing power from three different supercomputers on two continents, even though they are separate and have different hardware and software. The authors built a system that keeps track of data and training steps safely, handles sites joining and leaving, and chooses when to start training jobs to reduce waiting. Their system successfully trained a model across all sites together with some trade-offs but without losing data or duplicating work.
What this means in practice
- •For hpc system administrators: Pool fragmented computing allocations across multiple supercomputers to train language models without requiring shared storage or network connectivity.
- •For enterprise ai infrastructure teams: Run cross-facility large language model training on geographically distributed HPC resources with robust data handling and optimized scheduling.$Commercial implications: Enables multi-site AI model training services that aggregate compute across independent data centers to improve resource utilization and reduce training time.
Authors
Zarè Palanciyan, Thomas van Osch, Douwe van der Wal, Olivera Kotevska, Tim Kok
Abstract
Academic compute is fragmented: allocations are granted per facility, and facilities differ in accelerators and software stacks, schedule jobs independently, and share neither a network nor a filesystem. We present a system that pools such allocations to pre-train a single language model across three supercomputers on two continents, up to 7,400km apart: Snellius (NVIDIA H100), LUMI and Frontier (both AMD MI250X). It combines (i) DiLoCo-style two-loop training with an elastic, token-weighted Nesterov outer step for which zero, one or many live sites are all normal states; (ii) DARL, a data-leasing protocol whose heartbeat-backed leases guarantee that, within an epoch, no sample is trained twice or lost under crashes, late joins and work stealing; and (iii) queue-aware placement driven by unprivileged sbatch --test-only probes. Training Qwen3-0.6B on C4 for 20,000 optimizer steps, the three-site run reaches a held-out perplexity of 34.7, against 28.2 for a centralised baseline. Per-round overhead (weight exchange and checkpointing) stays near 110 s regardless of the number of local steps H, so its share of wall-clock time falls from 32% at H=100 to 5.7% at H=1,000 and 3.1% at H=2,000. In a 23.8 h three-site run with four site departures, 1.2% of granted data blocks were reclaimed and none was duplicated or lost. In an idealized queue-model projection, queue-aware placement shortens time-to-target by 18-43% compared with waiting for all sites to be allocated. Cross-site pre-training is thus operational rather than competitive: it turns fragmented allocations into one training run at a measured cost.