HiLO enables smooth performance drops for large AI systems under heavy load
HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework
Distributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
Summary
Large computer systems that use AI models sometimes get overloaded and slow down or stop working properly. The authors propose HiLO, a new system that helps these big services handle heavy usage by switching smoothly between different pre-tested modes where the quality drops predictably. This keeps the system working steadily without surprises, even during traffic spikes. HiLO has been tested at Microsoft in Bing, managing millions of users without big quality losses.
What this means in practice
- •For cloud service operators: Maintain predictable and testable product quality during high traffic by coordinating product-level load adaptations across AI-driven services.
- •For enterprise ai platform teams: Implement automated quality-aware load management that ensures stable AI model service quality without manual intervention under varying load conditions.
Authors
Knut Magne Risvik, Zhuowei Li, Nirupam Purushothama, Hunter Su, Greg Dunham, Yan Reznikov
Abstract
Graceful degradation is a key mechanism employed by large-scale systems to handle overload conditions. Its importance has grown further as GPU-intensive LLM inference introduces persistent capacity constraints that cannot be addressed through provisioning alone. Common degradation approaches rely on service-level mechanisms such as load shedding and circuit breakers, or on feature-level switches (some manually controlled) to preserve availability and quality. While effective at protecting services, such approaches lead to untestable product states and thereby unpredictable product quality. We present HiLO (High Load Optimizer), an automated product-level degradation system designed to handle graceful degradation at scale. HiLO maps observed load to a product state lattice: a small set of pre-validated product states, partially ordered by degradation severity. Dependencies are resolved at design time enabling deterministic and continuous control without requiring runtime dependency discovery. The same abstraction applies unchanged across heterogeneous services, including GPU-intensive deep models, LLMs and their serving layers. At Microsoft, HiLO is deployed in Bing (serving over a billion monthly active users), where it manages on average two traffic surges per week across all datacenters, bounding OneDCG (relevance) drop to under 0.5 points (the operational target) even under sustained load exceeding 2$\times$ baseline peak. HiLO complements existing service-level protections with product-level coordination and makes product quality measurable, testable and predictable.