Kernel policy switching speeds up language model server scheduling
AKTS: Sub-Microsecond Kernel Policy Switching for Language-Model Agents
Operating Systems
Summary
Language model servers need to handle many quick user requests and slower background tasks on the same computer system. The authors found that using one fixed strategy for scheduling processor time can't do both jobs well at the same time. They designed a system called AKTS that can quickly and safely switch between different scheduling policies inside the operating system. This helps the system respond fast when many users connect, but also keeps background work moving during quiet times, improving overall efficiency.
What this means in practice
- •For cloud infrastructure teams: Manage interactive and batch GPU workloads more efficiently by quickly switching kernel scheduling policies to optimize latency and throughput.
- •For operating system developers: Integrate fast, safe scheduling policy switches using verified eBPF code to enhance workload adaptivity on Linux-based systems.
Authors
Mohammadali Khodabandehlou, Mahdi Alizadeh
Abstract
GPU-backed LLM servers often multiplex interactive requests with background batch work on the same CPUs. During a request burst, the scheduler should protect time-to-first-token; between bursts, it should let background work make progress. A fixed kernel policy leaves one of these objectives on the table, so agentic OS control needs a way to switch scheduler behavior as the workload changes. The hard part is not deciding that a switch is useful, but applying it safely and fast enough for the kernel. Scheduler events occur every 1-10 $μ$s, and any code that runs there must satisfy the eBPF verifier. Scalar knobs are fast but expose only limited policy behavior, while generating new eBPF policy code is expressive but puts compilation, verification, loading, and possible verifier rejection on the runtime path. We present AKTS, which verifies a policy library once, at load time, and reduces the agent's runtime action to writing an integer index into an in-kernel array of preverified policies. An in-kernel tail call resolves that index. Because the agent emits an index rather than code, verifier failure is not a runtime outcome. On Linux 6.14, AKTS applies a policy switch in 920 ns (p50), matching scalar writes while switching whole policies; makes an invalid index inert across 60,217 invocations on an attached scheduler; and switches policies in a vLLM workload to capture 97% of a throughput policy's batch work while matching a latency policy's burst response.