Benchmark platform measures physical field model accuracy across patterns
PDE-OBS: Controlled Evaluation Across Observation Patterns
Machine Learning
Summary
Predicting physical fields like weather or fluids depends on where and how often you take measurements. The authors created PDE-OBS, a platform that tests how well different models predict these fields when the measurement locations and patterns change. They tested many models on thousands of scenarios and found that models trained for one pattern often do worse when tested on another. Training with mixed measurement patterns helped reduce errors, but models trained directly for a certain pattern usually worked best. This platform helps builders test and improve models under varying measurement conditions.
What this means in practice
- •For environmental modelers: Evaluate and improve forecasting models by testing performance under different spatial measurement layouts.
- •For sensor network designers: Design sensor layouts that maintain model accuracy when observation patterns vary in physical monitoring applications.
Authors
Ruichen Xu, Siyao Wang, Fang Wan, Jiacheng Qiu, Wenhan Gao, Jiaxing Zhang, Linsey Pang, Ravid Shwartz-Ziv, Yann LeCun, Yuefan Deng
Abstract
Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern does not characterize performance when that pattern changes. We introduce PDE-OBS, an integrated benchmarking platform spanning numerical data generation, model training, and inference and evaluation under varying observation conditions. It combines 560,000 fields and trajectories from seven partial differential equation families with configurable observation operators and seven adapted baseline methods for stationary reconstruction and short-horizon forecasting. Separating observation construction from physical records allows users to specify parameterized patterns and deterministic mixtures for training and testing while preserving prediction targets and data splits. The evaluation protocol uses references trained for each test pattern to compare models on identical test observations and targets, alongside equal-count groups for spatial-layout comparisons. On a 14,000-record subset, we evaluate 441 trained models under nine test patterns, yielding 3,969 evaluations. Mean cross-pattern error exceeds mean matched-pattern error in all 49 PDE-method pairs, and this finding persists in a configuration-matched subset of 117 models. Denser test observations do not consistently reduce error for a fixed model. Mixed-pattern training on five completed pairs reduces large single-pattern transfer errors, although destination-trained references usually remain more accurate. Together, the benchmark and findings support systematic evaluation of observation-pattern sensitivity and provide a reusable workflow for developing methods under changing measurement conditions. Code: https://github.com/ru1ch3n/PDE-OBS.