Team scaling boosts some small AI models more on math tasks
An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures
Artificial Intelligence
Summary
Using multiple smaller AI language models together can improve their accuracy, but whether adding more models helps depends a lot on the kind of task. The authors tested different ways to organize teams of AI models on various question types and found that math problems benefit the most from bigger teams, while other question types see little gain. They explain this by breaking down how much the teams generate possible answers and how well they choose and transform those answers. Their work shows that scaling AI teams is a strategic choice that works better for some tasks and setups than others.
What this means in practice
- •For ai system designers: Design AI teams with task-specific orchestration to improve arithmetic problem-solving accuracy efficiently.
- •For software developers: Choose AI model team architectures aware that adding calls may yield varying accuracy depending on the task type.
Authors
Blaz Bertalanic, Carolina Fortuna
Abstract
Replacing one LLM agent with a collaborating team can raise accuracy, but whether scaling the team helps, and which architecture to scale, is unclear. Sweeping eight agent orchestration architectures across five instruction-tuned 7-9B models, five short-answer benchmarks, and an executable-code benchmark up to 30 calls, we find that the returns to team scaling are sharply task-dependent: from three to thirty calls accuracy rises by up to 17 points on the two arithmetic word-problem benchmarks (GSM8K, GSMHard) but by at most four on ARC, GPQA, and MMLU, for every architecture, a split the usual task-averaged number conceals. Proposer-Critic captures the arithmetic gains, scaling steepest and, in aggregate, surpassing every other architecture at the largest budget (item-clustered intervals exclude zero), though it ranks among the weakest elsewhere, and no architecture wins across tasks. We explain these trajectories with an exact generate-transform decomposition. Partitioning any workflow into proposal coverage and a downstream transform, any accuracy change splits exactly into an extensive coverage dividend and an intensive transformation change. The decomposition diagnoses each task: arithmetic offers coverage headroom that a critic-guided transform converts, whereas the multiple-choice benchmarks either saturate in coverage or fail to convert it, and on open-ended code generative recovery nearly vanishes so accuracy tracks coverage. At equal call budgets token cost still varies 2.1x. Extra calls therefore create candidate opportunity that only some architectures, on some tasks, convert. Team scaling is a task- and architecture-specific bet, not a uniform lever.