MemX cuts time and balances memory in mixed GPU training clusters

MemX: Memory-Aware Automatic Parallelism for Cross-Vendor Heterogeneous GPU Training

Distributed, Parallel, and Cluster Computing

Summary

Training large AI models requires many GPUs, which can be different in power and memory. This makes it hard to split work fairly and efficiently. The authors created MemX, a system that finds the best way to divide tasks across different GPUs by considering both their memory and speed. MemX works much faster and more accurately than previous tools, even on bigger GPU clusters.

What this means in practice

  • For machine learning engineers: Optimize distributed training of large models on clusters with different GPU types by balancing compute and memory use efficiently.
  • For cloud infrastructure managers: Reduce search time and cost when configuring GPU clusters from multiple vendors for AI workloads by using MemX’s quick configuration search.

Authors

Lin Meng, Yuzhong Sun, Jie Zhu

Abstract

Cross-vendor heterogeneous GPU clusters are increasingly used for large-scale model training. However, their GPUs differ substantially in compute power, memory capacity, and interconnect characteristics. Existing automatic parallelism training systems usually minimize iteration time while treating memory as a hard constraint, almost failing in resolving the compute--memory asymmetry of cross-vendor clusters regardless correlation between memory and throughput. In this paper, we present MemX, a new memory-aware automatic parallelism system that jointly optimizes iteration time, peak memory usage, and cross-GPU memory balance. MemX consists of a weighted-product objective function, heterogeneous recomputation strategies, a minimal-set profiler, interpolation-based data augmentation, a dual-output XGBoost predictor, and a hash-signature prediction cache, in order to accelerate search of large-scale auto-parallel best configurations at lower cost. We implement MemX on Megatron-LM and evaluate it against four automatic parallelism baselines on two heterogeneous clusters. MemX achieves the lowest prediction Mean Absolute Percentage Error (MAPE) of 12.17% and is the only method that identifies the real Top-1 configuration under both time-based and memory-balance rankings. In terms of search cost and search space, MemX searches 31,240 configurations in 26.5 seconds, while an end-to-end profiling baseline is extrapolated to require about 17 days at the same scale. The simulation study that extends the measured 20-GPU data to 40-GPU and 60-GPU clusters further illustrates the effectiveness of MemX at larger scales.