Language model compute adjusts automatically based on latency and quality
Latency-Aware Adaptive Best-of-$N$
Computation and LanguageMachine Learning
Summary
Some language models improve answers by spending more computing time at use, but deciding how much extra effort to spend can be tricky, especially when servers get busy and responses slow down. The authors created a two-step method that first checks some answers and then uses how good they are and how long they took to decide if more answers should be generated. This approach changes its expectations based on how fast the server is working without needing to directly measure server load. Their tests show it gets similar quality answers while spending less computing effort on average, especially when latency varies a lot.
What this means in practice
- •For online service operators: Adjust computational effort on language model responses dynamically to reduce total user wait time and computing cost under varying server load.
- •For chatbot development teams: Improve chatbot responsiveness by adaptively choosing how many answer samples to generate based on response quality and latency without direct monitoring of server load.
Authors
Alon Jacobson, Qian Xie, Peter I. Frazier
Abstract
Test-time compute (TTC) improves language model responses by spending additional computation at inference time, and the right amount of compute depends on the prompt. Adaptive TTC methods allocate compute based on signals of response quality such as observed rewards, often by stopping once the signal exceeds a fixed threshold, and generally measure cost by the number of generations or tokens. But in online serving, latency is also a cost: it grows with server load, so a method using a fixed threshold spends too little compute when generation is fast and too much when it is slow. For Best-of-$N$ sampling, a widely used TTC method, we propose a two-stage adaptive algorithm that draws an initial batch, then uses the observed rewards and the batch's latency to decide whether to draw a second batch and at what size by comparing expected improvement against predicted cost: it draws one when the first batch's best reward is below a threshold that decreases with the first batch's latency, without explicitly measuring server load. We map each reward to its value under the cumulative distribution function (CDF) of the prompt's reward distribution, so the same tradeoff parameters can be reused across tasks; a regression fitted offline on other prompts predicts a distribution for the CDF value of the first batch's best reward, giving the expected improvement in closed form. We evaluate the method by simulation on a preference-based alignment benchmark, measuring generations-equivalent cost: generations plus latency, converted to generations. The method reaches Best-of-$N$'s reward at lower generations-equivalent cost, and conditioning on the measured first-batch latency saves at least as much of that cost as fixed-threshold (latency-unaware) baselines, and more when latency is highly variable, when it grows with batch size, and when the second batch's latency is highly predictable from the first's.