Orchestration platform enables fast deployment of data center network controls
CCAaaS: An Orchestration Platform for High-Velocity Design, Development, and Deployment of Data Center Transports at Meta
Networking and Internet Architecture
Summary
Data centers run different ways to manage traffic depending on the type of data being sent, like quick messages or large AI computations. The authors created a system called CCA-as-a-Service (CCAaaS) to quickly develop and update these traffic controls without stopping the data flow. This system uses a technology called eBPF to build flexible, fast, and efficient tools that monitor and manage network congestion. Using CCAaaS, Meta handles most of its server traffic smoothly and can improve network speed and reduce delays by a lot.
What this means in practice
- •For data center network engineers: Manage and update network congestion controls rapidly without disrupting ongoing traffic using a modular eBPF-based orchestration platform.
- •For cloud infrastructure operators: Observe and diagnose congestion control performance at scale with a lightweight tool integrated into production networks.
Authors
Prankur Gupta, Balasubramanian Madhavan, Miao Xu, Prashanth Kannan, Jie Meng, Neil Spring, Rajiv Krishnamurthy, Srikanth Sundaresan, Martin Lau, Theophilus A. Benson
Abstract
Today, data center networks run multiple congestion control algorithms (CCAs), each tailored to distinct traffic characteristics, from latency-sensitive intra-region RPCs to high-throughput cross-region AI workloads. The rapid evolution of AI training and inference further accelerates this trend by continually introducing new traffic characteristics that demand new CCAs. This paper introduces CCA-as-a-Service (CCAaaS), our eBPF-based ecosystem for the rapid design, deployment, and operation of production-grade CCAs at hyperscale. We present (1) a modular BPF-TCP framework and general lessons for designing modular CCAs; (2) a deployment and orchestration layer that provides disruption-free CCA updates, along with best practices for managing a zoo of CCAs at scale; and (3) a lightweight observability tool for CCAs and a blueprint for scaling CCA observability. CCAaaS manages over 97% of our server-to-server traffic using three active eBPF-based CCAs; two of the five CCAs we have built have since been deprecated. We show that our eBPF-centric approach to CCAs development delivers the same application-level performance (retransmissions, bandwidth, and tail latency) as their native in-kernel counterparts, and that our custom CCAs add negligible CPU and memory cost over their eBPF baselines. In terms of realized benefits, the CCA built with CCAaaS significantly improved network, and application metrics, for example, one of our CCAs reduced P99 latency for cross-region services by up to 20X.