Micro interventions reveal how language models will learn next
Target-Independent Micro-Interventions for Predicting Training Response Across Language-Model Families
Machine Learning
Summary
Benchmark tests show what a language model can do now but don’t tell us how it will improve with more training. The authors propose making small, target-free tweaks—called micro-interventions—that reveal how a model might respond to further training. These responses, combined with current abilities, help predict future learning across different families of language models more accurately. Their method works better than just relying on the model’s existing capabilities and generalizes across various models.
What this means in practice
- •For ml engineers: Predict how different language models will improve with additional training without relying on task-specific data.
- •For ai model deployment teams: Estimate performance gains from updating language models before committing resources to retraining.
Authors
Zhongxuan Liu, Sicheng Zhou, Hongzhi Wang
Abstract
Benchmark scores describe what a checkpoint can do now, but they do not determine how it will respond to the next training episode. We measure this missing state by branching four short, standardized, target-independent micro-interventions from the same checkpoint and recording their effects in a common capability space. Together with current capability, these responses form L-State; its pulse block supports a flexible direct readout and a structure-preserving operator readout. Under smooth local dynamics, the operator construction admits an end-to-end cross-family bound with explicit source- and target-family coordinate heterogeneity. In three-family leave-one-family-out development, both pulse readouts reduce source-standardized MSE by 39.4% relative to capability alone, while separating the best response and direction estimates. On sealed GLM-4-9B, the direct and operator readouts reduce MSE by 71.8% and 78.3%, respectively, and the operator readout raises sign balanced accuracy from 0.366 to 0.754. On sealed Granite-3.1-8B, the direct readout reaches RMSE 0.544 and a development-fitted action-wise selector reaches 0.554, compared with 1.172 for capability alone. A five-family audit finds that the operator coordinate varies by action and family, and that modeling these deviations improves retrospective held-trajectory prediction. Target-independent interventions therefore expose training-response information that current capability misses, with direct and structured readouts covering complementary transfer regimes.