Internal prediction gaps can grow large without harming final predictions
How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL
Machine Learning
Summary
This paper shows that even if two methods for updating beliefs in a model differ a lot internally, their predictions about future observations can still be very similar. The authors prove this using a mathematical example involving hidden Markov models with Gaussian noise. They explain that differences in internal updates may not always cause mistakes in tasks that rely on predictions. This finding helps clarify when simplifications or approximations inside prediction systems may still work well in practice.
What this means in practice
- •For machine learning engineers: Optimize belief update algorithms by allowing simpler approximations that maintain good predictive accuracy despite large internal differences.
- •For statistical model designers: Assess when model compression or approximate updates can be employed without sacrificing predictive performance.
A theory result. No direct application yet.
Authors
Qifu Wen, Shuaijun Liu, Zihan Zhou, Xi Zeng, Ningxin Su
Abstract
Accurate posterior prediction need not require accurate approximation of Bayesian updates. We prove that an unbounded gap between the update maps can coexist with vanishing predictive KL for every fixed finite $K\ge2$ in a stationary symmetric Gaussian HMM. Exact Bayesian mixing and an explicit deterministic radial filter act on the same $K-1$ belief coordinates. As $q\to0^+$, their separation in centered logits in the worst case grows at least linearly in the natural confidence scale $L_K(q)$, while their categorical $D_{\mathrm{KL}}(\mathrm{exact}\|\mathrm{radial})$ vanishes at the same explicit witness. Along stationary HMM trajectories, the expected terminal KL between filtered posteriors also converges to zero at $H(q)=\lceil-\log(q)/c\rceil+1$. Typical blocks without switches drive both filters into a common confidence cone, where softmax curvature suppresses their disagreement; a single Gaussian maximal event controls adaptive noise. A sweep with equally spaced Gaussians over $K\in\{2,4,8\}$ illustrates the opposing trends, and binary controls at long horizons compare saturating and nonsaturating recurrences. The result isolates two missing links between internal update gaps and predictive cost: the contribution of separating states to expected loss and decoder sensitivity. Thus even an unbounded internal update gap does not by itself certify predictive failure. The construction is fixed in $K$ and does not provide a universal criterion for when compression is harmless or characterize when internal gaps must incur task loss.