Learned critic scores weather forecasts by spotting unrealistic patterns
Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism
Machine Learning
Summary
Weather forecast models sometimes create blurry or unrealistic patterns even if their individual predictions seem accurate. The authors developed a system that learns to tell real weather data apart from model-generated forecasts, helping to detect these issues. This learned critic adapts to different types of errors automatically, unlike existing fixed tests that only catch specific problems. It successfully identified and ranked errors in simulated data and found that forecasts lose realism with longer prediction times. It also showed numerical weather models generally produce more realistic forecasts than machine learning ones.
What this means in practice
- •For weather model developers: Use the learned critic to identify and rank unrealistic errors automatically in new weather forecast models.
- •For atmospheric data analysts: Evaluate and compare realism of forecast outputs from different weather models over varying lead times.
Authors
Younes Elberkennou, Dmitri Demler, Thierry Meier, Luca Rispoli, Fanny Lehmann, Joel Oskarsson
Abstract
Despite their high accuracy on point-wise metrics, machine learning weather forecasting models can exhibit different failure modes such as blurring, periodic irregularities, and other unphysical spatial artifacts. This has motivated a variety of metrics to detect known failure cases. Existing metrics fix a representation or transformation in advance, and that choice limits the artifacts they can detect. We propose to train a discriminator for separating reference data from the model's output, and using its output logit to obtain a divergence-like realism score. The discriminator learns whatever separates the model's fields from real weather, adapting to whichever failure mode that model exhibits. We compare our learned atmospheric critic to existing metrics using various synthetic corruptions applied to ERA5 reanalysis data. Our method successfully identifies the corruptions and ranks their severity, while existing metrics fail on at least one corruption. Additionally, we evaluate forecasts from real weather models, and find that the realism score degrades with longer lead times and the metric generally assigns higher realism to numerical models than to machine learning models.