Adaptive router improves machine learning service quality in kubernetes
A Kubernetes-Native Request Router for Quality-Aware Inference Serving in the Computing Continuum
Distributed, Parallel, and Cluster Computing
Summary
Handling machine learning tasks across many computers can be tricky because you want answers to be fast and accurate. The authors created a system called ASRB that smartly decides which computer should answer each question, based on speed, accuracy, and other factors. ASRB can adjust to changing conditions without needing to change the main Kubernetes system. Tests show it can make responses faster or more accurate depending on settings, while also cutting down the work needed to keep track of everything.
What this means in practice
- •For cloud infrastructure engineers: Manage ML services across multiple computing resources with improved balance between speed and accuracy using ASRB's flexible routing policies.
- •For edge computing teams: Deploy adaptive request routing that reduces monitoring costs and improves service reliability on distributed edge devices running Kubernetes.
Authors
Ignjat Karanovic, Pantelis A. Frangoudis, Ivan Čilić, Ivana Podnar Žarko, Schahram Dustdar
Abstract
We introduce Adaptive Score-based Routing Balancer (ASRB), a dynamic, score-based request routing mechanism for Kubernetes-based service deployments over the computing continuum. ASRB jointly considers infrastructure-level information, response time measurements, and application-level quality indicators, with a particular focus on serving Machine Learning (ML) workloads. For these workloads, ASRB balances requests over service instances deployed in the continuum, following service provider-defined policies encoded as weighted combinations of QoS criteria to flexibly address latency-accuracy trade-offs. To drive routing decisions and swiftly adapt to changes in the operating environment, ASRB monitors a range of runtime metrics across multiple system layers. To deal with the associated monitoring overhead, particularly important for large-scale deployments, it selectively and adaptively controls monitoring intensity without sacrificing on routing quality. ASRB is implemented without requiring any modifications to Kubernetes, making it straightforward to deploy and operate in existing cluster environments. Our testbed experiments demonstrate the versatility of ASRB: When tuned for latency reduction, it achieves at least 10 ms lower mean response time compared with latency-oriented state-of-the-art routing mechanisms, while it achieves higher accuracy when this is prioritized through specific configurations, thus enabling flexible and operator-controllable trade-offs. At the same time, it attains reduced failure rates, higher responsiveness to changes in the operating environment, and up to ~70% less monitoring cost than relevant state-of-the-art solutions, at the potential expense of only a modest latency penalty in some configurations.