Data value depends on coverage not just individual scores
When Does Data Value Reduce to Class Balance? A Coverage View of Per-Point Data Valuation
Machine Learning
Summary
Deciding which data points to keep when training AI models is usually done by picking the top scorers individually. This paper shows that this approach fails when data points are very similar because their value depends on what other points are already kept. The authors explain when simply balancing the data by class matches smarter coverage methods, and when more complex coverage strategies are needed. They also provide a way to tell in advance which method to use based on how well the data’s classes align with the model’s understanding.
What this means in practice
- •For machine learning engineers: Select training data more effectively by balancing per-point value scores with class information to improve model accuracy especially when data is redundant.
- •For ml operations teams: Diagnose in advance whether class balancing or coverage-based data selection will yield better results for model updates under tight data budgets.
Authors
Yingjie Bai, Weiming Zhi
Abstract
Per-point data-valuation scores -- Data Shapley, influence, leverage -- are routinely used to select training data: keep the points with the top $k$ scores. We argue that top-$k$ by a fixed, mode-blind per-point score is the wrong selection rule precisely in the regime where data selection matters. The reason is that a point's value is not a fixed number but depends on what else has been selected -- a near-duplicate of an already-chosen point adds almost nothing, however high its score -- and a per-point scalar cannot express this interaction. The theory quantifies this failure exactly: in an equal latent-mode model, fixed top-$k$ selection incurs a closed-form excess risk over adaptive coverage because it repeatedly spends budget on already-covered modes. The gap is largest when the budget is tight, the learner representation is redundant, and missing a mode is costly. The practical rule is simple: use balance when the labels available at selection identify the learner's modes, and use explicit coverage when they do not. On standard classification tasks with strong pretrained features, class-balancing the same per-point scores recovers the missing coverage and matches or outperforms generic set coverage across our datasets. Without labels, label-free coverage improves accuracy by up to +0.20 while raw top-$k$ can fall below random; when labels are coarser than the modes, coverage within coarse strata performs best. We turn this distinction into a pre-specified diagnostic and validate it across both favorable and stress-test settings. The limits are part of the claim: the advantage is representation-relative. Coverage fails when the relevant modes are not expressed in the learner geometry, as with spurious subgroups, or when data quality varies independently of coverage -- exactly as the theory predicts.