Simplex lattice method improves merging of language model experts
SLIM: Simplex-Lattice Interpolation Merging
Machine Learning
Summary
Combining different experts in large language models to work well together takes many costly tests. The authors propose a method called SLIM that builds a simple quadratic model to predict how well different mixes will perform, using as few tests as possible. This model lets them find the best combination without testing every possibility. Experiments show it predicts results accurately and works better than random testing approaches when test budgets are limited.
What this means in practice
- •For machine learning engineers: Reduce costly evaluations needed when merging multiple specialized language model components into one efficient model.
- •For product data teams: Better combine expert model outputs with fewer tests to improve product features dependent on language models.
Authors
Seongcheol Jeong, Masahiro Suzuki, Yutaka Matsuo
Abstract
Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.