AI summaryⓘ
The authors studied how to adapt language models without changing all their weights by using gradient-free methods that randomly tweak weights and keep the best tweaks. They tested what really matters for these tweaks: how many weights to change, which directions to pick, and how big the changes are. Their experiments showed that changing just a small number of parameters performs almost as well as tweaking every weight, and surprisingly the main factor affecting success is the size (norm) of the change, not the subspace or specific directions chosen. This finding helps simplify how to efficiently adapt language models without full retraining. In short, the key is controlling how big the random weight perturbations are, rather than where exactly they happen.
language modelsparameter-efficient adaptationgradient-free optimizationweight perturbationsubspaceperturbation normmodel adaptationrandom searchSVDGrassmann overlap
Authors
Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim, Unggi Lee
Abstract
Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars. Gradient-free adaptation, which samples random weight perturbations and keeps the ones that score well, has not followed that trajectory and still perturbs every entry of the weight tensor. It is unknown whether that full-weight search is necessary, and more fundamentally which property of a perturbation makes it work at all, because existing methods vary the search space, the perturbation scale, and the aggregation together. We resolve this by intervening on one factor at a time inside a fixed pipeline, holding candidate scoring and voting constant while we vary the search dimension, the subspace that carries the perturbation, and its norm. Perturbing a frozen frame of 12 to 16 scalars stays 1.8 accuracy points behind full-weight search on average across 49 model-benchmark cells, trailing it in 36 of them. Neither the dimension nor the choice of basis explains that performance. A random frame whose Grassmann overlap with the SVD frame is at chance level performs identically once a single scale factor is matched, and at large scales the SVD directions collapse first. What survives is the perturbation norm, whose usable range closes within a factor of five across seven models and stays flat inside. The perturbation norm is therefore the one factor with a failure mode, and its safe region transfers across scale and family. The design question narrows from which subspace to perturb to how hard to shake.