Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
The authors study how large language models (LLMs) that share GPUs are usually handled one request at a time, which can cause delays especially when using tools that require continuous sessions. They introduce Talaria, a system that keeps track of ongoing sessions to better schedule model use and memory handling, reducing waiting times. By considering factors like the model's presence in memory and prior work done in a session, Talaria significantly speeds up processing times compared to traditional methods. Their experiments on large models show substantial improvements in how fast sessions complete.
Authors
Utopia Meng, Unicornt Zhao, Derek Li, Goalen Gao, Frank Du
Abstract
Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.