Patterns explain when frozen models adapt well to time changes
The Anatomy and Boundary of Adaptation under Temporal Tabular Shift
Artificial Intelligence
Summary
Models trained on tabular data can lose accuracy as data changes over time, and sometimes updating them helps but sometimes it hurts. The authors analyze why adaptation helps or harms by breaking down the problem into four key factors under a setup where each new label appears only after a prediction. They find limits on what can be learned from data without labels and identify conditions where adaptation is feasible. Their study uses theory and simulated data to understand when frozen models can adjust successfully to changing data streams.
What this means in practice
- •For machine learning engineers: Determine when updating frozen tabular models on streaming data will improve or worsen performance under temporal changes.
- •For data platform architects: Design monitoring tools to measure adaptation difficulty of tabular models operating on evolving industrial data streams.
Tested on simulated data.
Authors
Tianyu Wang, Xi Vincent Wang, Lihui Wang, Mian Li, Zhihao Liu
Abstract
Prequential adaptation of frozen tabular foundation models under temporal drift, with each label revealed only after prediction, helps some deployments and harms others, yet current practice does not predict which. We study the sources and limits of these gains. A diagnostic anatomy attributes gains to four recurring mechanisms under a streaming protocol that removes three optimistic biases and quantifies a fourth. Within an agnostic total-variation drift class, the target conditional is only partially identified: its identified-set diameter, the \emph{wall}, is irreducible from unlabeled data uniformly in sample size. A second, orthogonal $L^2$ projection wall quantifies what the frozen representation cannot express. Two canonical mechanism priors collapse the first wall. Under stated nuisance-rate conditions, the wall can be estimated from labeled historical windows at a $\sqrt N$ rate above the margin threshold $γ^\star=d_0/(2α_s)$. At $γ=0$, the conditional lower-bound program depends on an open affinity estimate; the positive-margin lower branch also remains open. Semi-synthetic data illustrate the finite-sample mechanism with calibrated exponents. Stream-level proxies on eight industrial streams fall on the difficult side under a stated roughness bound, while the equality case $γ=γ^\star$ remains unresolved.