Flow ensemble filter improves state estimates from partial noisy data
Improving Ensemble Filters with Flow Matching
Machine Learning
Summary
Estimating the true state of a system based on noisy or incomplete data is a common challenge. Classical methods use collections of guesses but assume simple, smooth updates that can limit accuracy. The researchers introduce a new method, Flow Ensemble Filter (FlowEF), that learns how to update each guess more flexibly by modeling complex transformations. This method better predicts states especially when observations are sparse and noisy compared to traditional ensemble filters and recent advanced models.
What this means in practice
- •For weather forecasters: Improve weather prediction accuracy by better integrating sparse and noisy sensor data into dynamical models using FlowEF updates.
- •For remote sensing data analysts: Enhance monitoring of environmental changes by more precise data assimilation from incomplete satellite observations with FlowEF.
Authors
Haoyuan Chen, Alexandre Thiéry
Abstract
Data assimilation estimates a dynamical state from partial and noisy observations. Classical ensemble filters are efficient but restrict analysis updates through finite sample covariance and affine Gaussian distribution. We introduce the Flow Ensemble Filter (FlowEF), which uses conditional flow matching to transport the forecast ensemble from a classical baseline filter to an analysis ensemble. FlowEF uses a localized Gaussian source during training, transports forecast ensemble members from a baseline filter at deployment, and conditions its velocity field on ensembles from that baseline filter and the observation. The proposed model therefore learns a nonlinear update while mapping each baseline ensemble independently. For sparsely observed dynamical systems, FlowEF improves both deterministic and probabilistic metrics over all four classical ensemble filters. It also achieves the best performance among the state-of-the-art generative data assimilation models.