Gradient data attribution methods track answer format more than task
Form Over Content In Gradient-Based Data Attribution Methods
Computation and LanguageArtificial Intelligence
Summary
Some methods that look at how training data influences AI models focus on comparing gradients, or small changes in the model. This paper finds that these methods often measure how similar the answer formats are (like whether answers are multiple choice or free text), rather than the actual meaning or task the data addresses. The authors tested this by changing answer formats independently from tasks and saw that gradients aligned mainly with format similarity. This finding suggests caution in using gradient-based methods to understand what AI models learn about tasks.
What this means in practice
- •For machine learning engineers: Avoid misinterpreting gradient-based data attributions by checking answer format effects when selecting training data for language model fine-tuning.
- •For ai model auditors: Evaluate the robustness of gradient attribution explanations by testing on datasets with varied answer formats and tasks to prevent misleading insights.
Authors
Sunwoo Kim, Seokwon Jung, Sohyung Kim, Seong Joon Oh, Alice Oh
Abstract
Data attribution methods using gradient similarity are widely used to analyze and select training data for large language models, but what gradient similarity actually measures is debated. Some interpret it as identifying task-relevant skills, while other work reports that surface form is the main factor. We resolve this debate for supervised fine-tuning examples by varying task and answer format independently. Specifically, we render benchmarks in different answer formats, such that datasets can share a task without a format or a format without a task. We find that gradient alignment follows the answer format, as benchmark pairs sharing an answer format align strongly (disattenuated cosine near 0.4), while same benchmarks rendered with different answer format classes show no alignment (near 0.0). We demonstrate that this ordering holds from the earliest pretraining checkpoints through post-training, and across model scales and families. We then analyze the released selections of LESS, a gradient-based data selection method for instruction tuning, and find that each target's selections over-represent the target's own answer format. Hence, we demonstrate that gradient-based attribution methods track format similarity more than task semantics, meaning that such methods, as well as the semantic interpretation of the gradient, should be tested on data where answer format and task vary independently for greater robustness and reliability.