Benchmark creates standard tests for machine learning in power systems
EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection
Machine Learning
Summary
Power systems need protection to stop problems like blackouts, but testing machine learning methods for this is hard because people use different data and tests. The authors made EvEMTBench, a shared and open set of tasks and data that everyone can use to compare their methods fairly. It covers different electrical grids and situations, showing that machine learning models don’t always improve with more data or when conditions change. This benchmark helps researchers and engineers find out where their models work well or fail, making evaluations clearer and more reliable.
What this means in practice
- •For power system engineers: Use the benchmark to compare machine learning methods for detecting and locating faults under realistic and changing grid conditions.
- •For electric utility operators: Evaluate protection systems across multiple voltage levels to improve fault response strategies and grid reliability.
Authors
Julian Oelhaf, Georg Kordowich, Christian Bergler, Andreas Maier, Johann Jäger, Siming Bayer
Abstract
Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.