Precipitation forecasts can mislead when trained on wet-biased data
Wet-Biased Data Pipelines Mislead the Training and Evaluation of Generative Precipitation Nowcasting
Machine Learning
Summary
Training artificial intelligence models to predict rain often uses datasets with too many rainy examples. The authors show this can make the models learn and be tested on an unrealistic view of weather, causing evaluations to be misleading. When tested on data that matches real-world conditions, the same model can seem to predict too much rain instead of too little. This means forecast developers might be led in the wrong direction when improving models based on biased tests.
What this means in practice
- •For weather forecasting teams: Avoid misleading evaluation of rainfall forecasts by verifying models on deployment-like balanced datasets instead of wet-biased samples.
- •For climate risk analysts: Improve understanding of model biases when using AI precipitation forecasts, supporting better risk assessments for extreme weather planning.
Authors
Christoph Metzl, Stefan Kapsreiter, Kianusch Vahid Yousefnia, Virginia Poli, Richard Müller, Frank Krüger, Tobias Bölle
Abstract
Generative models for precipitation nowcasting are trained and evaluated predominantly on wet-biased datasets, which over-represent precipitation events to emphasize rare extremes. Selecting test cases based on target observations is known to favor forecasts that match the selected distribution rather than the intended deployment distribution. We show analytically how this target-dependent sampling also shifts the conditional distribution learned by generative models. When training and test sampling match, even an ideal model learns precisely the distribution favored by evaluation, allowing the evaluation bias to conceal the training bias. Using a fixed satellite-to-radar diffusion model, we demonstrate that target-dependent test sampling can reverse the calibration assessment. The model appears under-predictive on a wet-biased test set but over-predictive on a deployment-representative test set. This result is replicated outside the model's training domain after a seven-year gap. Utilizing classifier-free guidance as an example, we demonstrate how the contradictory calibration assessment can misdirect downstream model development.