Evolving benchmark reveals where environmental models fail over time
EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling
Artificial IntelligenceMachine Learning
Summary
Environmental models predict things like streamflows and climate, but they often fail under changing conditions, new locations, or rare events like floods. The authors created EvoBench, a tool that shows exactly when and where different models struggle instead of just giving one average score. They tested this on daily streamflow predictions from thousands of U.S. basins and found no single model is best in all situations. EvoBench also helps identify and fix model weaknesses by tracking errors as they change through time.
What this means in practice
- •For environmental modelers: Detect and track how well models perform across different times, locations, and rare conditions to improve prediction reliability.
- •For water resource managers: Better identify and anticipate weaknesses in streamflow predictions for improved flood and drought planning.
Authors
Yiming Sun, Yuan Zhang, Qi Cheng, Yingda Fan, Runlong Yu, Yiqun Xie, Xiaowei Jia
Abstract
Machine learning models are now widely used to predict environmental variable dynamics in domains like hydrology, climate science, agriculture, and ecology. These models are usually compared on fixed benchmark datasets using a single average accuracy score. As models improve, their scores become very close when averaged over large regions and long time periods, so the benchmark can no longer tell them apart, and a single number hides \emph{where} and \emph{when} a model fails. Three failures are common across environmental domains: models get worse as the climate shifts (non-stationarity), models fail at unmonitored sites (spatial extrapolation), and models miss rare but important regimes such as floods and droughts. We present \textbf{EvoBench}, a benchmark framework that shows where a model fails, tracks failures over time, and evolves to focus on model capability gaps. We apply EvoBench to daily streamflow prediction as a case study (\textbf{CAMELS-Evolve}, about 9{,}000 U.S. basins, 1985--2024) and compare an operational physical model (NWM), a conceptual reference (HBV), and several data-driven models. No model wins on every testing scenarios (e.g., in terms of average accuracy, performance at low flows and unmonitored basins). Also we observe that all data-driven models under-predict floods. Additionally, we show that targeted fix based on the reported failures could mitigate weaknesses the benchmark identified in certain scenarios. EvoBench turns model evaluation from a saturated ranking into a moving map of model limitations.