Estimating label noise in machine learning with guaranteed accuracy
Estimation of the Label-Noise Transition Matrix with Performance Guarantees via Selective Classification
Machine Learning
Summary
Machine learning often uses large datasets with labels that can be incorrect, which harms learning accuracy. The authors propose a new way to estimate how labels get confused without needing fragile guesswork techniques, and their method can guarantee its performance even with limited data. This approach uses selective classification that focuses on certain confident predictions, making the estimation more reliable and flexible. They also provide efficient algorithms and prove mathematically how well their methods work.
What this means in practice
- •For machine learning engineers: Improve training on noisy labeled data by accurately estimating label errors with reliable performance guarantees.
- •For data annotation teams: Use improved estimation methods to identify and correct label noise in large annotated datasets before model training.
Authors
Xabier de Juan, Santiago Mazuelas, Yilun Zhu, Clayton Scott
Abstract
Modern machine learning depends heavily on massive datasets, but obtaining high-quality annotations at scale is often expensive. As a result, learning from noisily-labeled data has become common, making accurate estimation of the label-noise transition matrix crucial. However, existing transition matrix estimators rely on the fragile estimation of class-posteriors and do not provide finite-sample performance guarantees. In this work, we propose a novel methodology to estimate the transition matrix based on one-sided selective classification. This approach bypasses class-posterior estimation, provides finite-sample performance guarantees, and leverages flexible learning methods for binary classification. Moreover, we introduce effective algorithms to implement the proposed methodology and provide their refined finite-sample performance bounds.