Enterprise AI gateway manages multiple large language models smoothly
Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceSoftware Engineering
Summary
Large organizations often use many different AI language models that each have different ways to connect and pay for services. The authors designed a system called Cosmos LLM Gateway to handle this complexity by acting as a single entry point that manages use, costs, and security policies. This system efficiently routes requests, applies rules to protect data, and keeps track of spending while working across many AI providers. It supports many developers and business uses with very high reliability and low delay.
What this means in practice
- •For enterprise ai teams: Run multiple commercial large language models together with unified access, spending limits, and data protection in one scalable gateway.
- •For cloud infrastructure engineers: Integrate a single policy-driven proxy to handle latency-sensitive AI request routing and authorization while minimizing overhead in production.
Authors
Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav TK, Ashwin Hegde, Anand Gautam
Abstract
Large organizations consume large language models (LLMs) from several vendors at once, each with its own API dialect, credential model, streaming protocol and pricing. Integrated directly, this heterogeneity yields credential sprawl, format lock-in, unbounded spend and inconsistent governance. We present the design and production operation of the Cosmos LLM Gateway, the governed entry point for enterprise LLM traffic at PayPal. Built on Envoy, it splits work by latency profile: I/O-bound control decisions (authentication, credential brokering, policy and budget evaluation) run in an out-of-process external-processing service, while wire-format translation, token metering and prompt-cache shaping run in-proxy as a Rust WebAssembly filter, so that, by default, streamed response body chunks never wait on an out-of-process call. Models, providers and prices are largely validated data rather than code. Governance is declarative policy evaluated by Open Policy Agent and managed through a control plane with role-scoped authoring, approvals, pilots, shadow-mode rollout, usage notifications and versioned policy bundles. For use cases that require it, a data-loss-prevention layer can block, redact or pseudonymize detected sensitive content before it reaches a provider. Spend scopes are views over per-model usage counters, so a new budget applies to spend already recorded in its window without migration. As of September 2026, the gateway serves more than 12,000 developers and hundreds of business use cases, handling roughly 200 billion tokens per day across more than 100 models at 99.999% availability to business. In a benchmark of the core request path against a synthetic upstream, it adds a median of 4 ms and a p99 of 11 ms of overhead at more than 2,200 requests per second on two 4-vCPU instances. We report the architecture, policy model, evaluation and lessons learned.