ExaServe enables large scale AI model serving on supercomputers
ExaServe: Large-Scale LLM Serving on Exascale HPC Systems
Distributed, Parallel, and Cluster Computing
Summary
Running large language models (LLMs) on supercomputers is hard because it requires lots of special setup and tuning. The authors created ExaServe, a tool that makes deploying these big models on supercomputers easier by using simple config files. They tested it on a powerful supercomputer named Aurora, showing it scales well up to hundreds of nodes for certain types of requests. However, they found some performance limits and delays when managing many nodes. This work helps make LLM deployment on huge computers more practical while pointing out future challenges.
What this means in practice
- •For hpc operators: Deploy and manage large language model services efficiently on supercomputing clusters using reproducible configurations.
- •For cloud infrastructure engineers: Optimize large-scale AI serving frameworks by identifying bottlenecks in control-plane design at multi-node deployments.
Authors
Wenyi Wang, Shu Shi, Yadu Babuji, Ian Foster, Kyle Chard
Abstract
Cloud-native LLM serving frameworks have made deployment routine in data centers, yet deploying them on leadership-class supercomputers remains an engineering challenge requiring scheduler integration, MPI launch, accelerator selection, node-local weight staging, and platform-specific patches. We present \textit{ExaServe}, a pip-installable framework that transforms a declarative YAML specification into a reproducible large-scale LLM serving deployment. Using ExaServe, we deploy LLM serving on ALCF Aurora from 1 to 256 nodes (3072 vLLM replicas). Non-streaming inference scales nearly linearly to 256 nodes, reaching 27.1\,k requests/s (3.8\,M tokens/s). Token streaming scales differently: a centralized proxy plateaus at $\sim$4.7\,k requests/s despite the model servers remaining within the service-level objective. We also identify an \emph{O}(\emph{N}\textsuperscript{2}) Ray Serve control-plane bottleneck that increases cluster bring-up to $\sim$30 minutes at 256 nodes. ExaServe provides a practical, reproducible deployment path while exposing key barriers to future exascale LLM serving.