Latent space filtering improves forecasting with sparse noisy data
LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations
Machine Learning
Summary
Forecasting how complex systems change is hard because simulations take a lot of time and observations are limited and noisy. The authors developed a new method called LD-EnFF that works in a simpler, compressed version of the system, called a latent space, to speed up forecasting and make better guesses. LD-EnFF uses a special neural network to understand how observations relate to this latent space and updates its predictions more efficiently. Tests show it outperforms many existing methods on problems like fluid flow, tsunamis, and weather with sparse and noisy data.
What this means in practice
- •For weather forecasting teams: Improve atmospheric state estimation from limited and noisy sensor data using faster latent space ensemble filtering.
- •For disaster monitoring centers: Enhance tsunami propagation predictions with efficient updates from sparse sensor measurements.
Authors
Ziyu Tian, Kaichen Shen, Wenbo Hao, Phillip Si, Peng Chen, Wei Zhu
Abstract
Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.