Zeroth-order method selects training data for large language models efficiently
ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
Machine LearningArtificial Intelligence
Summary
Influence functions help pick important training examples for big language models, but traditional methods need lots of memory and slow computations. The authors propose ZOIF, a new approach that estimates influence using only forward passes through the model, avoiding expensive memory use and backpropagation. Their method breaks the process into three steps that rely on approximations to efficiently calculate influence scores. Experiments show ZOIF picks useful data subsets while using far less memory and similar running time compared to prior methods.
What this means in practice
- •For machine learning engineers: Select key training samples for updating large language models efficiently on limited hardware using only forward computations.
- •For software debugging teams: Identify influential training examples that affect model outputs to help debug large language models without heavy memory demands.
Authors
Lisong He, Yunxuan Jiang, Haishan Ye, Xiangyu Chang
Abstract
Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.