Platform learns to find lowest cost reliable language model providers
Learning the Cost of Reliable Inference
Artificial IntelligenceComputer Science and Game TheoryMachine Learning
Summary
Many large language model services charge the same fixed price for each word they generate, which might not be the best deal for every task. The authors created a system that lets different language model providers bid for each user query, encouraging them to offer true prices based on their real costs. Over time, the system learns which providers deliver good quality at the best prices and routes user requests accordingly. They tested this idea using popular language models on math and question-answering tasks and found a big difference in prices, showing current fixed pricing can be very inefficient.
What this means in practice
- •For api platform operators: Implement competitive bidding among language model providers to route queries for cost and quality optimization in real-time user requests.
- •For cloud service buyers: Access language model services priced dynamically based on competition, reducing costs while ensuring quality for specific workloads.
Authors
Dimitrios Rontogiannis, Ander Artola Velasco, Manuel Gomez Rodriguez
Abstract
Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on these platforms typically use a fixed price per token, preventing users from achieving the most competitive price for their tasks. % workloads. In this work, we design a procurement platform where token prices for each task are driven by provider competition, enabling users to secure competitive pricing for guaranteed quality levels. To this end, the platform sequentially routes queries via a reverse second-price auction that incentivizes model providers to truthfully bid their best estimate of the average cost to serve a user's query. As it routes queries, the platform learns the quality offered by each provider and progressively routes queries to the most cost-competitive provider among those meeting a desired quality threshold. To validate our design, we conduct experiments with multiple LLMs from the \texttt{Llama} and \texttt{Qwen} families on popular mathematical reasoning and question-answering benchmarks. The results show that the pricing margin of the most cost-competitive provider on our platform varies significantly---from $10\%$ to $71\%$---depending on the task and quality threshold. This suggests a substantial inefficiency in the current fixed-price market, and it demonstrates that our platform may enable users to capture maximum savings whenever competitive market conditions permit.