Agentic AI staleness driven by refresh schedule not cache age
ChurnBench: A Drift-Aware Benchmark Demonstrating That Refresh Scheduling, Not Cache Age, Governs Staleness in Agentic AI
Software Engineering
Summary
Many AI systems answer questions using data that changes often, like prices or user info. The authors created ChurnBench, a new tool that tracks changes over time so it can tell if an AI's answer was once correct but became outdated. They found that how often the data is refreshed, not how old the stored data is, controls whether answers become stale. This helps developers focus on refresh timing to keep AI answers accurate.
What this means in practice
- •For enterprise it teams: Use ChurnBench to evaluate and optimize data refresh schedules for AI answering enterprise data questions accurately over time.
- •For customer support platform developers: Improve AI response freshness in systems retrieving customer data by testing different refresh strategies with ChurnBench.
Authors
Vivek Kumar Singh, Preeti Priyam
Abstract
In production, agentic systems answer questions over data that lives in several places and keeps changing: licenses are reassigned, users offboarded, prices changed, contracts renewed. Existing retrieval benchmarks freeze the data, so they cannot ask whether an agent's answer is still true, only whether it found the right passage. We present ChurnBench, an open-source benchmark that generates a four-source enterprise data fabric as a timeline rather than a snapshot. Every change is written to an append-only ground-truth ledger, and gold answers are computed from that ledger, never from the live stores. An answer that was correct when its data was retrieved but wrong when evaluated is therefore detected and labeled a freshness error, distinct from a reasoning error; we validate this by resolving ground truth at both timestamps for every case reported. Using the instrument, we find that when a system refreshes on a schedule, cache age does not predict staleness. Across cache ages of 1, 14, and 28 days, freshness errors were 7, 4, and 4, because scheduled refresh bounds staleness by time-to-live, and no TTL lapse was observed in any window. A controlled ablation confirms the mechanism: disabling tiered refresh raises freshness errors from 4 to 45 at 28 days and leaves them identical at one day. The variable a drift benchmark should sweep is therefore TTL configuration against each entity's rate of change, not drift-window length. ChurnBench, the evaluation harness, and all per-error data are released open source.