Grouper schedules cores to speed up multi-tenant microservices
Grouper: Scheduling Groups for Multi-Tenant Microsecond-Scale Microservices
Operating SystemsDistributed, Parallel, and Cluster Computing
Summary
Microservices often share computing cores with batch tasks, but switching cores between tasks can slow down requests significantly, especially when many users share the system. The authors introduce Grouper, which groups runtimes to let services donate cores directly to each other during request calls. This reduces the overhead of core allocation, making responses faster even with many tenants and multiple service hops. Grouper showed much better performance in tests compared to previous schedulers and Linux.
What this means in practice
- •For cloud platform engineers: Improve scaling latency-critical microservices running alongside batch jobs by using Grouper's grouping of runtimes for more efficient core scheduling.
- •For data center operators: Increase overall batch job throughput without degrading latency-sensitive workloads by adopting Grouper’s core donation approach in multi-tenant environments.
Authors
Koosha Kazemi, Mohammad Siavashi, Ahmad Siavashi, Mohammad Izadi
Abstract
Microsecond-scale core allocation makes colocating latency-critical services with batch work worthwhile. A thread that finds no work parks within microseconds and its core goes to a batch task. Putting one back costs $\sim$18 $μ$s, as the allocator must discover that a core is wanted and then take it from the batch task holding it. A monolith pays that tax once per request, a microservice chain pays it at every hop in both directions, and a multi-tenant host multiplies it again, because every tenant's hops queue at the same allocator. On our port of DeathStarBench's hotelReservation, going from two tenants to ten takes a hop from 39 to 222 $μ$s and a 10-RPC path's median from 456 to 2,445 $μ$s, a fivefold degradation even though no tenant's own load changed. We introduce Grouper and the scheduling group, a set of isolated runtimes that the allocator treats as one allocation and accounting unit, whose members may hand cores directly to one another. A service sending an RPC donates its core to the peer through an unprivileged kernel fast path, so the core follows the request through the call graph. The allocator retains control through reconciliation, core-addressed revocation and a pooled budget but leaves the critical path; its load falls from $Θ(R \cdot H)$ to $Θ(R)$ in request rate $R$ and hop count $H$. Over a grid of two to ten tenants at 1,000-30,000 requests per second each, Grouper outperforms Caladan (the allocator Junction also builds on) and Linux by up to 7.9$\times$ and 3.4$\times$ at the median and 4.1$\times$ and 14.2$\times$ at the tail, and leaves batch work more throughput than Caladan at over 70% of load points.