Combining multiple smaller language models improves large model reasoning accuracy
Multiple Importance Sampling for Guided Speculative Inference
Machine LearningInformation Theory
Summary
The paper looks at how to improve the way large language models (LLMs) solve problems by using several smaller LLMs to suggest possible reasoning steps. These smaller models each come up with ideas, and then a bigger model checks and scores these ideas to find the best answers. The authors use a mathematical method called multiple importance sampling to combine the suggestions in a way that balances their contributions. They show this approach keeps performance strong while avoiding problems when one model’s suggestions don’t fully cover the needed reasoning steps.
What this means in practice
- •For nlp engineers: Improve reasoning accuracy in LLM-powered applications by integrating diverse smaller models for proposal generation combined with a large model verifier.
- •For cloud ai platform teams: Optimize test-time inference cost and robustness by distributing work across several smaller LLMs rather than relying on a single large model drafter.
Authors
Joseph Rowan, Ashish J. Khisti
Abstract
We study an extension of the Guided Speculative Inference (GSI) algorithm for test-time-scaling of large language models (LLMs) to a novel setting where an ensemble of smaller LLMs proposes draft reasoning steps which are later verified by a larger base model and a process reward model. Using the established framework of multiple importance sampling (MIS), we introduce weighting schemes to aggregate samples from a diverse collection of proposal distributions. We provide new bounds on the KL divergence from the optimal reward-tilted target distribution and the expected reward degradation, then demonstrate that the well-known balance heuristic is optimal. Compared to GSI with a single well-tuned drafter, combining several diverse LLMs while retaining the same total number of samples maintains competitive pass-at-$N$ downstream performance while being mathematically robust to situations where the support of a single draft model does not cover that of the target, where GSI may fail.