FailureAtlas: A Taxonomy of Failure Modes in Multi-Provider LLM Serving Infrastructure
2026-07-20 • Machine Learning
Machine LearningSoftware Engineering
AI summaryⓘ
The authors study how multi-provider LLM gateways—systems that manage and balance requests to different language model APIs—can fail in different ways. They created a framework called to categorize these failures by where they start and how obvious they are (loud or silent). Using real bug reports and testing, they found several failure types, including some silent ones that look fine on the surface but actually cause serious problems. Two major silent failures the authors found involve lost conversation history and corrupted data from streaming issues, which are hard to detect without advanced monitoring.
LLM gatewaysreverse proxyfoundation modelsfailure modesnetwork layersession stateconcurrency race conditionstreaming protocolssemantic observabilityhealth checks
Authors
Vishal Pandey, Gopal Singh
Abstract
Multi-provider LLM gateways reverse proxies that route, load-balance, and rate-limit requests across foundation-model APIs have become critical production infrastructure. Yet the failure modes specific to this architectural layer remain undocumented, scattered across issue trackers and post-mortems with no unifying framework. We introduce \fa{}, a two-axis taxonomy that classifies failures by their \emph{origin layer} (Network/Transport, Streaming/Protocol, State/Session, Model~Behavior, Governance/Cost) and their \emph{detectability} (Loud vs.\ Silent). We populate this taxonomy with five verified catalog entries sourced from public bug reports and first-hand stress testing, each accompanied by a mechanistic root-cause analysis. Three entries include standalone reproduction scripts. Our principal finding is that the most operationally severe failures are \emph{silent}: they return HTTP~200, pass every standard health check, and corrupt application state in ways that require semantic-level observability to detect. Two such silent failures a concurrency race condition causing history loss and a streaming index collision corrupting tool-call payloads were discovered first-hand during \cb{} evaluation campaigns.