New way to measure how extra training data helps machine learning
Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach
Machine Learning
Summary
Sometimes, machine learning models get extra helpful information during training that they can't use later. This paper offers a new way to measure how much that extra information might improve a model’s accuracy overall. Instead of focusing on specific algorithms, the authors use a general math approach to estimate the maximum benefit that extra data can provide. They also introduce an easy-to-calculate metric that predicts these benefits from training data alone. Their method works for different kinds of learning problems, helping decide if the extra data is worth using before testing the model.
What this means in practice
- •For machine learning engineers: Estimate the maximum benefit of extra training-only features to decide whether to include them before model deployment.
- •For data science teams: Prioritize data collection by quantifying the potential gain from collecting privileged information alongside standard features during training.
Authors
Vasily Bokov, Sebastian Schmitt, Vedran Dunjko, Hao Wang
Abstract
In practice, various learning scenarios provide access to auxiliary features exclusively during training. Incorporating such data to enhance model performance gave rise to a paradigm known as Learning Using Privileged Information (LUPI). While this extra information is intended to improve the resulting model, establishing a generalized, cohesive understanding of how privileged information (PI) transfers useful knowledge remains a challenge. Vapnik's original theory and subsequent works offer performance guarantees in certain cases, but these results are inherently per-algorithm and rely on setting-specific proof approaches. Consequently, a more general framework explaining how and when PI transfers useful knowledge is still missing. To bridge this gap, we introduce an algorithm-agnostic, information-theoretic approach based on the PAC-Bayes framework. Rather than asking whether a particular algorithm exploits PI, we ask how much value it could offer: comparing the tightest achievable risk bound with and without PI yields its potential - an upper limit on the extractable gain. We introduce a metric that quantifies this potential directly from empirical training risk, bypassing the need for test-time data access, and validate our findings in both supervised and unsupervised settings. The results demonstrate a robust correspondence between our training-time metric and true test-time performance gains. Ultimately, this work takes a necessary step toward an information-theoretic understanding of LUPI, and quantifying the potential of privileged features before committing to a model.