AI summaryⓘ
The authors address the problem that deep neural networks often give high confidence scores even when they are wrong, making it hard to trust their predictions. They propose a new method called TCP_α that creates clearer differences in confidence between right and wrong predictions by adding a controlled penalty to mistakes. This method guarantees a clear margin separating correct from incorrect confidence values, regardless of how many classes there are. They also study how to effectively train models on this kind of unbalanced data and show that their approach improves reliability in two different music-related tasks, even when the data changes. Overall, their method helps better identify when the model might be wrong and improves performance when uncertain predictions are rejected or when adapting to new data.
deep neural networksconfidence estimationpost-hoc calibrationclassification errorsdecision boundaryimbalance learningdomain shiftregression problemrāga identificationfailure prediction
Authors
Parampreet Singh, Anushka Singh, Sumit Kumar, Vipul Arora
Abstract
Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions. Consequently, users lack a reliable signal for deciding when a prediction can be trusted. Post-hoc confidence estimation addresses this by training a lightweight auxiliary head over a frozen classifier. Existing targets, however, suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions. In this work, we propose $TCP_α$, a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples. We prove that $TCP_α$ guarantees complete separation between the target values of correct and incorrect predictions, with a separation margin that is independent of the number of classes and increases monotonically with the penalty parameter. Since accurate classifiers naturally produce very few errors, learning these targets results in a severely imbalanced regression problem. We therefore present a systematic study of training strategies for learning under this imbalance and identify an effective training configuration through extensive ablation studies. We evaluate the proposed approach on rāga identification, investigate its robustness under domain shift, and further validate it on frame-wise ornamentation detection without modifying the selected configuration. Across all settings, $TCP_α$ consistently outperforms existing confidence targets for failure prediction. Rejecting only the least-confident 8\% of predictions improves the base model's macro-F1 from 0.89 to 0.98, while fine-tuning the confidence head with only 5\% labeled samples from a new corpus effectively restores performance under domain shift.