HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters
2026-07-03 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors looked at training large AI models that can handle both text and images, which is harder and slower compared to text-only models. They found reasons why the training wasn't efficient and created a system called Mpipe to fix this by better organizing the steps involved. Mpipe uses a method to rearrange tasks so the computer hardware is used more fully, speeding up the training process. Tests on specific hardware showed Mpipe made training up to almost three times faster on small setups and still improved speed on very large ones.
multimodal large language modelsMFU (machine fraction utilization)decoder-only LLMschedule algebraheterogeneous parallel schedulepipeline parallelismAscend 910C NPUmodality encoderlarge-scale trainingspeedup
Authors
Chong Li, Zhengdao Yu, Nelson Lossing, Thibaut Tachon, Pierre Leca, Etienne Filhol, Yujie Yuan, Chong Bao, Teng Su
Abstract
Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70x speedup in a small-scale setting and 1.21x speedup in a 512-card large-scale setting.