Micro interventions predict how language models improve during training
Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Families
Machine Learning
Summary
Benchmark scores show what a language model can do at a point in time but don't tell us how it will improve after more training. The authors introduce quick tests called micro-interventions to measure how a model is likely to respond to further training, independent of any specific task. They combine these interventions into a new state called L-State that better predicts training improvements across different model families. Their methods significantly reduce errors in predicting training response compared to using capability alone, showing that these micro-interventions capture important information missed by conventional benchmarks.
language modelbenchmark scoretraining responsemicro-interventioncapabilitycheckpointmean squared errortransfer learningmodel familyreadout
Authors
Zhongxuan Liu, Sicheng Zhou, Hongzhi Wang
Abstract
Benchmark scores describe what a checkpoint can do now, but they do not determine how it will respond to the next training episode. We measure this missing state by branching four short, standardized, target-independent micro-interventions from the same checkpoint and recording their effects in a common capability space. Together with current capability, these responses form L-State; its pulse block supports a flexible direct readout and a structure-preserving operator readout. Under smooth local dynamics, the operator construction admits an end-to-end cross-family bound with explicit source- and target-family coordinate heterogeneity. In three-family leave-one-family-out development, both pulse readouts reduce source-standardized MSE by 39.4% relative to capability alone, while separating the best response and direction estimates. On sealed GLM-4-9B, the direct and operator readouts reduce MSE by 71.8% and 78.3%, respectively, and the operator readout raises sign balanced accuracy from 0.366 to 0.754. On sealed Granite-3.1-8B, the direct readout reaches RMSE 0.544 and a development-fitted action-wise selector reaches 0.554, compared with 1.172 for capability alone. A five-family audit finds that the operator coordinate varies by action and family, and that modeling these deviations improves retrospective held-trajectory prediction. Target-independent interventions therefore expose training-response information that current capability misses, with direct and structured readouts covering complementary transfer regimes.