Audio feature masking often changes output beyond expected signal effects
Mask-Induced Displacement in Audio XAI via Logit Trajectory Decomposition
SoundMachine Learning
Summary
When analyzing which parts of an audio signal matter for decisions, people often hide parts and see how the result changes. This paper shows that how you fill the hidden parts matters a lot and changes the output in ways not just related to the retained sound. The authors break down these changes into two parts and find that a big chunk of the change comes from unexpected directions. This finding means that current explanation methods overlook important effects caused by the fill and could be improved by accounting for these.
What this means in practice
- •For audio machine learning engineers: Improve explanation tools for audio models by correcting biases from masking fills that affect output unexpectedly.
- •For speech recognition developers: Refine model debugging methods by separating masking fill effects to better identify true signal importance.
Authors
Nico García-Peguinho, David Kelly, Fabrizio Smeraldi, Anna Xambó Sedó
Abstract
Perturbation-based XAI methods for audio classifiers often estimate feature importance by masking spectrogram regions and crediting output changes to the retained signal. Yet they typically assume the fill (the mask replacement) is negligible. We propose logit-space trajectory decomposition to examine this assumption. An on-axis component captures output along a line connecting the fully filled (occluded) spectrogram to the fully retained original; an off-axis component captures perpendicular displacement. We evaluate across three fills, three audio classifiers, and 1,100 AudioSet clips. We demonstrate that no fill is acoustically neutral under full occlusion: Zero fill activates silence and Gaussian Noise fill activates broadband noise. Under partial masking, 41-77% of output displacement is off-axis, with the direction of the residual stable across mask retention fraction and specific to each model-fill combination. Attribution methods are unevenly exposed to off-axis displacement through their sampling and weighting strategies, revealing apparatus-dependence. Where the off-axis residual is stable and low-dimensional, its structure affords mitigation.