Machine learning fault detectors show more false alarms across relay locations
Characterizing Cross-Relay Generalization Failure in Machine Learning-Based Power System Protection
Machine Learning
Summary
Machine learning models used to detect electrical faults often train on data from specific locations in a power grid, but measurements vary across different spots. The authors studied how well a fault detector trained on one relay’s data performs when applied to others. They found that overall accuracy looks stable, but false alarms increase a lot when applying the model to different relay locations. This means measure-specific differences can hide problems that appear when moving machine learning models across the network.
What this means in practice
- •For power grid operators: Safely deploy machine learning fault detectors on one relay location and assess false alarm risks when transferring models across other relay sites in the grid.
- •For electrical protection engineers: Improve design and evaluation of fault detection algorithms by separately tracking false positive and false negative errors across different relay locations using local measurements.
Tested on simulated data.
Authors
Nithin Pradeep Nadayil Kizhakkethil, Julian Oelhaf, Christian Bergler, Andreas Maier, Siming Bayer
Abstract
Machine learning-based fault detectors are often evaluated at the relay locations used for training, although local voltage and current measurements vary across power networks. This work investigates cross-relay generalization for binary detection of faults anywhere in the network using local measurements, rather than line-selective tripping. Using 50 ms windows from a simulated 110 kV transmission network, we evaluate a multilayer perceptron across all 64 source target combinations of eight relays and five random seeds. A shared simulation-level split and source-only threshold selection provide a controlled held-out evaluation without target-specific calibration. Despite nearly unchanged mean F1-scores (0.730 within-relay versus 0.734 cross-relay), the mean false-positive rate (FPR) increases from 0.088 to 0.341, while the mean false-negative rate decreases from 0.399 to 0.318. The change is strongly directional: models trained at Bus 2 relays yield a mean FPR of 0.670 at Bus 1 and Bus 3 relays, compared with 0.096 for cross-relay transfers within the same bus group. Linear probing indicates that learned representations retain coarse bus-group information, while exploratory class-conditional latent distribution analysis shows no consistent monotonic association with transfer errors. These findings demonstrate that aggregate metrics can conceal location-structured changes in false-alarm behavior, highlighting the importance of evaluating error types separately when assessing cross-relay generalization.