Scalable data influence estimation improves model predictions and interpretation
Data Attribution at Scale via Influence Matrix Estimation
Machine Learning
Summary
Understanding how each training example affects a machine learning model’s predictions is important but computationally expensive, especially for neural networks. The authors look at this problem and find a way to estimate a big matrix that captures these effects using fewer calculations. They introduce two new methods that work efficiently with existing tools and show better results on different sizes of data and compute budgets. This helps make data attribution more practical for tasks like understanding models or removing training data without retraining.
What this means in practice
- •For machine learning engineers: Estimate training data effects on predictions more efficiently to improve debugging and interpretability of neural network models.
- •For data privacy teams: Enable faster data removal impact assessment to support machine unlearning efforts in compliance with user data deletion requests.
Authors
Yuxi Chen, Hamza Golubovic, Han Tong, Arian Maleki, Andrew Ilyas
Abstract
Data attribution seeks to quantify how individual training examples shape a model's predictions and underpins problems including data valuation, machine unlearning, and model interpretability. Despite having a long line of work, computationally scalable methods often struggle to predict the effect of removing training data in neural networks due to their non-convex nature. To overcome this challenge, metagradient-based methods such as MAGIC (Ilyas and Engstrom, 2025) differentiate each prediction through the entire training run and compute its exact influence with respect to the training data, but require a separate run for every prediction. To reduce this cost, we cast budgeted attribution as estimating a large influence matrix from a small number of measurements. We show that the measurements most appropriate for recovering this matrix differ from those best suited for attribution itself. We then present two algorithms, MAGE and SPELL, suited for reconstruction and attribution respectively, that run on existing metagradient machinery at no extra cost. Empirical studies demonstrate strong performance over existing baselines across training scales and measurement budgets.