Contextual priors change confidence in brain speech decoding errors
Confidence-Ordering Reversal under Contextual Priors in Neural Decoding
Artificial IntelligenceInformation Retrieval
Summary
Decoding brain signals to recognize speech often uses context to improve accuracy, but this can make some wrong guesses seem more confident than they really are. The authors found that after adding context, some errors with bigger confidence margins are actually less likely to be fixed, depending on where the correct answers rank initially. They studied this effect using brain data and proposed a way to keep track of both local cues and context separately to give better confidence estimates.
What this means in practice
- •For brain signal processing teams: Improve confidence estimates in neural speech decoding systems by separating local and contextual evidence, enhancing reliability of decoded brain signals.
- •For speech recognition developers: Refine error detection in speech systems that combine neural data and language models by accounting for confidence-ordering reversals in fused scores.
Authors
Xinyu Zhang, Sichao Liu
Abstract
Contextual priors improve neural-to-language decoding by reshaping candidate scores. However, confidence is read from the same reshaped scores, so the errors a prior leaves behind can become more confident with no change in accuracy to reveal it. We study how a prior shapes confidence in speech retrieval on MEG-MASC and MOUS using local decoding scores, a contextual prior combined by additive shallow fusion, and the fused top-two margin as confidence. Among initially incorrect predictions, we find a confidence-ordering reversal: a larger margin makes a repair more likely when the correct candidate starts near the top of the local ranking, but less likely when it starts lower. On MEG-MASC, pooled correctness AUROC is 0.87, yet AUROC separating repairs from residual errors falls from 0.70 at initial ranks 2-3 to 0.39 at ranks 21-50. Errors starting beyond rank 20, inside the reversed region, make up 46.6% of all post-fusion errors. We propose a score-level account: a repair must first close the correct candidate's initial deficit, limiting its final margin, whereas a residual error can build a large margin between two incorrect candidates. A causal intervention that changes only the fusion weight moves the reversal to deeper ranks as predicted. Under a word-level LM prior, it keeps moving after accuracy gain peaks, so a weight chosen for accuracy does not settle confidence. Reading local and prior scores separately improves selective decoding: the decoder answers on 74.5% of windows instead of 56.7%, while 92% of output sets still contain the correct candidate. Confidence after contextual fusion should retain the local and contextual evidence behind each prediction, not just the fused scores. Project website: https://confidencereversal.github.io/; Code: https://github.com/AmadeusFake/NeuDecodingConfReversal