Splitting training across many adapters improves voting accuracy in AI models
Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It
Machine Learning
Summary
When AI models try to answer questions multiple times and vote on the best answer, training a single model deeply can cause all answers to make the same mistakes. The authors show that training many smaller models (called adapters) separately and then voting among them produces better results than concentrating all training into one model. This approach, called an adapter thicket, keeps the benefits of training while avoiding correlated mistakes that lower voting accuracy. So, spreading training effort widely rather than deeply helps get more reliable answers.
What this means in practice
- •For machine learning engineers: Improve ensemble decision accuracy by dividing training into multiple smaller adapters rather than training a single large one.
- •For ai service operators: Increase reliability of AI-generated answers at scale by using adapter thickets to reduce correlated errors in majority voting.
Authors
Jonathan Williams, Esin Tureci Karthik R. Narasimhan
Abstract
Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy so that its samples increasingly make the same mistakes. With every method drawing exactly $160$ completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test ($1.5$B-$8$B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to $4.8$ points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to $7.0$ points. The cause is concentration, not RLVR itself. We split the same data and training budget across $K$ LoRA adapters, each trained on its own random disjoint shard, and call the result an adapter thicket. Thickets out-vote the fully trained adapter in all $16$ (model, $K$) settings, and for $K{\geq}4$ they stay within $0.8$ points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small $K$. For $K{\geq}8$, thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from $16$ to $160$ votes, the thicket's lead over the fully trained adapter widens from $1.3$ to $3.3$ points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.