Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
2026-04-24 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to efficiently plan experiments for large-scale training runs, where testing all options is too expensive. They treat choosing experiments as a smart budgeting problem, aiming to pick runs that best predict performance in costly scenarios. Their method uses uncertainty to guide which experiments to run next, focusing on the most informative ones. When tested across various tasks, their approach performs better than traditional methods and gets close to using the entire budget while only spending about 10%. This helps save money and time in planning big training experiments.
scaling lawsexperimental designbudget allocationuncertainty estimationsequential experimentsextrapolationmachine learning trainingcost optimization
Authors
Sijie Li, Shanda Li, Haowei Lin, Weiwei Sun, Ameet Talwalkar, Yiming Yang
Abstract
Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.