Efficient model assignment method balances quality cost and speed
Which Model for Which Role? Cost-Efficient Pareto Recommendation for LLM Agentic Workflows
Machine LearningComputation and Language
Summary
Different parts of a large language model (LLM) workflow may use different models based on what each part needs. Choosing which model to use where can be tricky because some models cost more to run but give better results. The authors propose a method named CC-Gittins that helps find the best possible combinations of models, balancing quality, cost, and sometimes speed, while reducing the number of tests needed. They tested it on various workflow scenarios and found it identifies good options using only a small fraction of the time and expense that a full search would require.
What this means in practice
- •For software engineering teams: Select cost-effective model setups for AI workflows that balance performance and compute expenses efficiently.
- •For cloud service providers: Optimize configurations of multi-model AI services to lower operational costs without compromising quality for users.
Authors
Qian Xie, Wenyue Hua, Yueli He, Sripad Karne, Armaan Agrawal, Nikos Pagonas, Eugene Wu, Kostis Kaffes, Tianyi Peng, Nairen Cao
Abstract
LLM agentic workflows increasingly employ different models for specialized roles. Users may prefer different trade-offs between task quality and deployment cost, sometimes also considering latency. Finding good assignments incurs a separate search cost from evaluating multiple candidates. We introduce Cost-Coupled Gittins (CC-Gittins), a multi-objective extension of single-objective cost-aware Gittins search for Pareto recommendation. It exploits the coupling between evaluation and deployment costs: assignments that cost more to deploy also cost more to evaluate. The search alternates between seeking higher quality and lower deployment cost, reusing each evaluation to inform both searches. After each batch, CC-Gittins recommends assignments that are nondominated under uncertainty-adjusted estimates of quality and deployment cost. We evaluate CC-Gittins on eight quality--cost matrices from workflows with two to five roles and up to 1,681 candidate assignments. We also evaluate the method with latency as a third objective. Across the eight benchmarks, CC-Gittins recovers a substantial fraction of the true Pareto frontier at roughly 10% of the exhaustive-evaluation cost, and recovers most of the frontier with few false positives by 30%.