RDMA software boosts large-scale AI model speed and efficiency
Deploying RDMA for Large-Scale Inference and Training Workloads
Machine LearningDistributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
Summary
Communicating data between computers in large AI systems is often slow and inefficient. The authors report on ByteExpress, a software layer that improves this communication by better managing memory and connections over RDMA hardware. Their system reduces delays and CPU use in both AI training and inference tasks, making the whole process faster and more efficient. This shows that smart software design is essential to fully benefit from advanced networking hardware in AI applications.
What this means in practice
- •For machine learning engineers: Improve speed and reduce processor load when training large AI models over multiple servers using RDMA hardware with ByteExpress software.
- •For data center network operators: Manage server-to-server communication more reliably and scalably in AI inference workloads by coordinating connections and recovery above RDMA transport layers.
Authors
Yibo Huang, Yi Cui, Yukai Huang, Huaping Zhou, Jiancong Tong, Zhaoyu Wu, Gen Miao, Haoran Wei, Boyan Ding, Zhuo Jiang, Jianxi Ye, Yibo Zhu, Chuanxiong Guo, Yang Xu, Jie Wu
Abstract
RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.