Adaptive score filtering improves high-dimensional nonlinear data assimilation
AECSF: Adaptive Ensemble Conditional Score Filtering for High-Dimensional Nonlinear Data Assimilation
Machine Learning
Summary
Estimating the true state of complex systems with many variables is hard because current methods either oversimplify or become unreliable. The authors propose AECSF, a new way to better estimate these states without needing training data. AECSF uses a clever statistical identity to improve accuracy by adaptively weighting multiple guesses about the system’s state. Their approach helps better capture uncertainty and improves predictions in challenging, high-dimensional problems with limited data.
What this means in practice
- •For weather forecasting teams: Improve weather prediction accuracy by better estimating complex atmospheric states with limited ensemble forecasts.
- •For financial risk modeling teams: Enhance nonlinear filtering of high-dimensional market data to produce more reliable risk estimates from sparse observations.
Authors
Yangwen Zhang, Shiwei Ni, Xiaoping Zhang, Xiaofei Guan, Lili Ju
Abstract
Bayesian state estimation for high-dimensional nonlinear dynamical systems entails a fundamental tension between statistical fidelity and computational tractability, as particle weights can collapse, while Gaussian ensemble updates can miss non-Gaussian posterior structure. Score-based diffusion filters offer a sampling-based alternative, but existing training-free score filters often rely on heuristic likelihood corrections, which can compromise posterior accuracy by neglecting uncertainty about the system state associated with each noisy reverse particle. To address these issues, we propose AECSF, a training-free adaptive ensemble conditional score filter. AECSF constructs an analytically tractable score estimator from the conditional Tweedie identity, which recasts noisy posterior score estimation as estimating the conditional mean of the system state given a noisy reverse particle and the observation. To estimate these conditional means efficiently, AECSF employs a shared adaptive weighted proposal ensemble, while particle-specific conditional weights yield an estimate for each noisy reverse particle without separate proposal sampling. The proposal ensemble is updated using reverse-particle information within the same reverse-diffusion run to improve conditional-mean estimation. Theoretically, we characterize when a fixed weighted proposal measure yields the exact noisy posterior score. Under stated assumptions, we establish a bound relating conditional-mean estimation errors to reverse-sampling endpoint error. Numerical experiments demonstrate that AECSF improves the accuracy of posterior sampling and nonlinear filtering in high-dimensional problems with limited forecast ensembles.