AI summaryⓘ
The authors study how statistical monitors that correct machine learning models while they run perform when data isn’t perfectly random (exchangeable). They focus on a technique called conformal test martingales, which guarantee low false alarms if data is exchangeable, but this is often not true in real scenarios, especially with dependent data and feedback loops. In their experiments, these guarantees fail on real time-series data, causing the correction mechanism to react too much and worsen the model temporarily. They suggest adding extra controls to detect such failures and propose a method to reduce harmful reactions without tuning for specific datasets. Overall, the authors highlight limitations in current anytime-valid inference methods when applied to dependent data and emphasize the need for careful monitoring.
Machine LearningStatistical MonitoringAnytime-Valid InferenceConformal Test MartingalesVille's InequalityExchangeabilityDependent DataKalman FilterTime-Series ForecastingNull Calibration
Abstract
Machine learning systems are increasingly corrected while they run, and the decision of when to intervene is increasingly delegated to statistical monitors. Anytime-valid inference promises evidence that can be acted on at any moment, exactly the guarantee this setting needs, and it is moving from theory into deployed monitoring. Conformal test martingales are the change-detection instrument, and Ville's inequality caps their false-alarm probability on exchangeable data. The guarantee is conditional. A deployment inherits it only if the stream it monitors behaves exchangeably. The premise is hardest to satisfy where these monitors are most useful, on dependent data and inside loops where the monitor modifies the learner whose scores it reads. It is also rarely measured. We measure it in a pre-specified case study, where such a monitor gates the online updates of a Kalman adapter correcting frozen time-series foundation models on five forecasting streams. On exchangeable synthetic streams, the same implementation fires in at most 1 of 60 runs. On the real streams, at alpha = 0.05, 135 of 135 clean-stream runs fired. The construction does not explain the firing; the failure comes from the deployed score stream itself. Repeated fires hold the gate's drift response active, and the gated filter amplifies the very transient it was designed to prevent. The component worth keeping makes no validity claim. Huber-style gating of the filter's own updates cuts isolated-spike degradation by an order of magnitude with no dataset specific tuning. Anytime-valid methods proposed for dependent data should therefore be accompanied by null-calibration controls and mechanism traces.