ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing
Machine Learning
Summary
The authors study how switching between different large language models (LLMs) during outages can make conversations lose their memory, hurting the user experience. They introduce two new ways to measure how well conversation history is kept when switching providers. To fix this, they build a smart system that remembers and forwards conversation history across different LLM services without losing context. Testing their method shows it preserves conversation continuity about 99% of the time, while old methods barely keep any. They also analyze timing issues during failover to avoid making problems worse by retrying too aggressively.
Authors
Vishal Pandey, Gopal Singh
Abstract
In production large language model (LLM) deployments, high API availability guarantees do not equate to conversational continuity. When a primary provider experiences an outage or strict rate-limiting, naive stateless failover mechanisms successfully maintain uptime but silently discard conversation history, severely disrupting the user experience. To rigorously quantify and resolve this failure mode, we introduce two novel metrics: Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO). We propose a stateful, multi-provider proxy architecture utilizing a History-Forwarding strategy to seamlessly reconstruct conversational state across heterogeneous LLM endpoints during failover events. Furthermore, we release continuity-bench, https://github.com/Vishal-sys-code/continuity-bench, an open evaluation harness designed to stress-test context preservation under high-concurrency provider failure conditions. Our empirical evaluation ($N=750$ failover events) demonstrates that our stateful proxy achieves a 99.20\% CPR [95\% CI: 98.27\%, 99.63\%], cleanly transferring deep conversational context to fallback providers, compared to a near-0\% preservation rate for standard stateless architectures. Finally, we characterize failover latency distributions, identifying the critical necessity of asynchronous exponential backoff with jitter to prevent cascading retry storms against strict-limit fallback APIs. Our results provide a principled foundation for building robust, state-preserving multi-model inference systems.