Papers for

hpc operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ExaServe enables large scale AI model serving on supercomputers

ExaServe: Large-Scale LLM Serving on Exascale HPC Systems

Abstract: Cloud-native LLM serving frameworks have made deployment routine in data centers, yet deploying them on leadership-class supercomputers remains an engineering challenge requiring scheduler integration, MPI launch, accelerator selection, node-local weight staging, and platform-specific patches. We present \textit{ExaServe}, a pip-installable framework that transforms a declarative YAML specification into a reproducible large-scale LLM serving deployment. Using ExaServe, we deploy LLM serving on ALCF Aurora from 1 to 256 nodes (3072 vLLM replicas). Non-streaming inference scales nearly linearly to 256 nodes, reaching 27.1\,k requests/s (3.8\,M tokens/s). Token streaming scales differently: a centralized proxy plateaus at $\sim$4.7\,k requests/s despite the model servers remaining within the service-level objective. We also identify an \emph{O}(\emph{N}\textsuperscript{2}) Ray Serve control-plane bottleneck that increases cluster bring-up to $\sim$30 minutes at 256 nodes. ExaServe provides a practical, reproducible deployment path while exposing key barriers to future exascale LLM serving.

Wed 9 SeptDistributed, Parallel, and Cluster Computing
The gist
Running large language models (LLMs) on supercomputers is hard because it requires lots of special setup and tuning. The authors created ExaServe, a tool that makes deploying these big models on supercomputers easier by using simple config files. They tested it on a powerful supercomputer named Aurora, showing it scales well up to hundreds of nodes for certain types of requests. However, they found some performance limits and delays when managing many nodes. This work helps make LLM deployment on huge computers more practical while pointing out future challenges.
Open → 2609.10812v1