GearUp cuts serverless AI delays by hiding load time in task waits
GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading
Distributed, Parallel, and Cluster ComputingMachine LearningOperating Systems
Summary
Running AI tasks on serverless systems can be slow because each task needs time to load important data before it starts. The paper’s authors made GearUp, a system that cleverly uses the waiting times between tasks to load this data early without wasting resources. GearUp also keeps ready-to-go task copies around to avoid slow starts when many requests arrive suddenly. This approach cuts how long AI workflows take and uses memory more efficiently.
What this means in practice
- •For cloud platform engineers: Reduce response time and resource waste for machine learning inference workflows on serverless platforms by scheduling model loading within existing task wait times.
- •For ai service operators: Improve throughput and latency handling for real-time AI inference services under bursty user requests by adaptively keeping loaded GPU instances alive.
Authors
Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang
Abstract
Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.