Bridging Agent Semantics with Spot Capacity: An Elastic and Recoverable Service Model
Distributed, Parallel, and Cluster Computing
Summary
The authors present SemSpot, a new way for large language model (LLM) services to handle tasks with varying urgency and complexity by using short-term offers based on price and likelihood of completion. This approach lets agents pick the best option according to the current task needs, making better use of cheaper spot capacity on cloud platforms. They studied common task patterns from benchmarks and showed how SemSpot can change costs and response times. The authors also propose extending this idea to keep track of partial results within longer requests to improve efficiency.
Authors
Minchen Yu
Abstract
LLM agents increasingly drive long-running cloud inference workloads in which model calls differ in urgency, redundancy, completion semantics, and replay cost. Model-as-a-Service (MaaS) platforms expose several service models for trading cost against latency, availability, and capacity commitment. These models operate primarily at request, job, or endpoint scopes and provide limited support for combining transient platform supply with the evolving semantics of an agent task. We present SemSpot, a semantics-aware service model that allows agent applications to leverage the spot capacity of LLM inference platforms. At the request level, SemSpot lets a provider publish short-lived offers over successful price, completion probability, and failure-notification deadline; the agent runtime selects among these offers using the current task state and completion rule. An audit of 1,535 cases from six agent benchmarks identifies four recurring workflow structures and shows how this service model may produce different cost, service-time, and fallback behavior. With specialized MaaS support, token-level SemSpot further preserves provider inference state and runtime-verified semantic segments inside a long request. We develop the service model, economic boundary, and the cross-layer research agenda required to realize SemSpot.