Learning how to weight training data improves AI model accuracy
High-Value Training Data from Noisy, Diverse Sources
Artificial IntelligenceMachine Learning
Summary
Training AI models well requires lots of data, but some data might be messy or not quite right for the task. The authors developed a way for models to learn which data examples are more helpful and should be paid more attention to during training. They do this by using a small set of trusted data to teach the model how to weight other data better. This method helps improve performance, especially when using mixed or noisy data sources like human feedback and synthetic data together.
What this means in practice
- •For machine learning engineers: Improve model training by automatically weighting diverse and noisy data to better match target domains.
- •For ai product teams: Combine human and synthetic feedback more effectively to build better reward-based AI models.
Authors
Hedi Driss, Evan Frick, Anastasios N. Angelopoulos, Cho-Jui Hsieh, Yejin Huh, Tijana Zrnic, Andrew Ilyas
Abstract
Training state-of-the-art models increasingly requires learning from large, heterogeneous datasets whose examples vary in quality, domain relevance, and source of supervision. Dataweighting---assigning different weights to training examples---offers a general way to control how such data influences training, but learning effective weights at scale remains challenging. We develop a scalable approach to learning dataweights: given a small, high-quality target dataset, we use metagradients through the training process to efficiently optimize training-example weights directly for performance on the target domain. To scale beyond a fixed dataset and model, we learn parametric weighting rules that generalize to new and streaming examples and transfer from smaller models to substantially larger ones. We study reward modeling as a prototypical application, where preference data may be noisy, adversarial, synthetic, or mismatched with the target domain. Learned dataweights improve held-out performance, automatically adapt reward models to specialized domains, and allow mixtures of human and synthetic supervision to outperform either source alone.