Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes
2026-07-06 • Operating Systems
Operating SystemsArtificial IntelligencePerformance
AI summaryⓘ
The authors tackle the problem of running large language model (LLM) decoding on-device using multiple CPU cores, which is tricky because all cores need to work together without interruptions. They introduce a system called Anima OS that lets the set of CPU cores working on the task change safely between tokens without causing errors or deadlocks. Their approach uses a special handshake protocol so cores join or leave cleanly, letting other tasks run on unused cores efficiently. Tested on real hardware, their method improves overall system performance compared to fixed core allocations, adjusting dynamically to workload demands.
LLM decodingCPU coresbarrier synchronizationelastic gang schedulingseqlockepoch protocolRCU (Read-Copy-Update)preemptive schedulerAnima OSthroughput
Authors
Daeyeon Son
Abstract
On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.