Graph model explanations warn of failing predictions before errors appear
Does the Explanation Drift Before the Guarantee? Label-Free Early Warning on Evolving Graphs
Artificial Intelligence
Summary
Models that analyze graphs can start making mistakes as the graph changes over time without obvious signs. The authors show that tracking how the model explains its own decisions can warn early that its predictions are becoming unreliable. They fix some technical issues with existing measures to better detect these warning signs. Their method can predict failures several steps ahead during gradual changes, while not raising false alarms when things are stable. However, it may miss some types of changes, so they use a second measure to cover those cases.
What this means in practice
- •For data engineers: Detect upcoming failures in graph-based models monitoring network or transaction data before prediction accuracy breaks down.
- •For cybersecurity analysts: Identify early warning signs in evolving network graphs indicating potential security model degradation prior to detection failures.
Authors
Tomasz Szandała, Damian Raczkowski
Abstract
Deployed models can fail silently as the world they describe evolves. Attribution drift has served as an early warning in tabular data; we ask whether, in graph streams, a model's own explanations can forewarn the loss of nominal conformal coverage. Doing so requires repairing the naive drift statistic twice. A vanishing edge loses mask mass regardless of whether the model relied on it, so we condition the importance on edge presence. Evolving graphs also break edge identity: a relabelling that changes nothing shifts the identity-based statistic by 0.69 on synthetic graphs and 0.83 on real ego-networks, whereas aggregating over structural roles brings both to 0.0 and stays well-defined when the support changes. We test six pre-registered regimes on thirty seeds each. A two-channel monitor, tracking conditional mask mass and its distribution over roles, precedes coverage collapse by 8 to 14.5 steps across four gradual regimes (node addition, removal, replacement, and edge thinning), all surviving Holm correction. A sudden knockout yields the expected zero lead, and a harmless change triggers no alarm. Leads track the length of the transitional phase (Spearman 0.79 over 150 runs). On real transaction ego-networks, the untuned mass channel forewarns staged drift by 27 steps and is the only channel, attributional or structural, that never alarms on a stationary stream. The main limitation is that the mass channel is blind to covariate drift, which breaks the guarantee in every run; the distributional channel detects it and also rescues the rewiring regime, so we report the two channels separately rather than as a union.