Architecture aware ensembling boosts model accuracy with fewer samples
Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbations
Machine Learning
Summary
When trying to improve AI language models by making small random changes and combining the best results, previous methods treated all parts of the model equally. The authors show that considering the unique characteristics of different parts of the model while making these random changes helps find better versions using fewer tries. This approach leads to more accurate results on various language tasks and works well across different model sizes. The findings highlight that how you make these random changes matters a lot when looking for improvements without using gradient methods.
What this means in practice
- •For machine learning engineers: Improve language model ensembles with fewer candidate models by using architecture-aware perturbations to speed evaluation and deployment.
- •For ai model deployment teams: Reduce computational costs in selecting and combining language model variants for production systems without losing accuracy.
Authors
Kirato Yoshihara, Hiroaki Hamade
Abstract
RandOpt samples weight-perturbed language models and ensembles top-ranked candidates through plurality voting, but its global perturbation scale ignores heterogeneous module geometry. We propose \mbox{\textbf{\emph{Modular Norm RandOpt}}}, an architecture-aware sampling method using module-wise natural norms and calibrated scales while preserving selection and voting. It outperforms RandOpt using $3\times$ fewer candidates on Countdown and at least $12\times$ fewer on GSM8K, with corresponding wall-clock savings. Evaluations across seven tasks and three Qwen scales ($0.5$B--$3$B) show higher mean accuracy than RandOpt on Countdown, GSM8K, and MATH-500 at every scale. The gains extend to Llama 3.2 $3$B and Gemma 3 $4$B on Countdown and GSM8K. On Qwen2.5-1.5B, our ensembles also achieve higher mean accuracy than iterative baselines on both tasks at comparable main-run evaluation budgets. On GSM8K, a tail-density diagnostic implies only a $1.2$--$1.8\times$ candidate reduction, while most ensemble improvement is associated with more favorable correct-expert support. These results highlight perturbation geometry as a key design choice for population-efficient, gradient-free search around pretrained models.