Optimal decoding for large language models uses tilted temperature sampling
The Optimal Decoder is Tilted Temperature
Machine Learning
Summary
Choosing the right temperature in language model sampling affects how well generated outputs perform, especially as you take more samples. The authors figured out the best way to adjust this 'temperature' by balancing two things: keeping enough variety so outputs don't all look the same, and making sure outputs are plausible. They developed a formula called tilted temperature that adapts based on how likely a partial output is to succeed. They also created methods to estimate this formula either during generation or from past data, improving code generation accuracy over standard techniques.
What this means in practice
- •For code generation engineers: Use tilted temperature decoding to boost code correctness and diversity in automatic code generation systems.
- •For natural language generation developers: Implement calibrated temperature decoding to improve output quality when sampling from large language models with diverse sample sizes.
- •For automated content generation platforms: Integrate optimal temperature strategies to enhance textual output reliability and diversity for commercial applications.$Commercial implications: It enables more reliable and diverse language model outputs, improving content generation products for businesses.
Authors
Nikolaos Papagiannis, Ananth Grama, Wojciech Szpankowski
Abstract
Larger sample budgets benefit from higher sampling temperatures in LLMs, but what temperature tuning optimizes and why it works remain poorly understood. We seek to answer these questions: we pose decoding as pass@$k$ maximization subject to two constraints that encode critical decoding failure modes: an entropy floor preventing diversity collapse, and a cross-entropy ceiling preventing implausibility. We prove that this maximization takes the form of a single closed-form conditional sampling rule, the \emph{tilted temperature} $q^\star \propto \bigl(p e^{ηV}\bigr)^{1/T}$. Here, the tilt comes from a verifier potential $V$ that quantifies how likely a partial generation is to eventually pass. Ordinary temperature sampling, verifier-greedy decoding, and reward-tilted policies represent special cases. We further show that, under a natural rule for scaling the diversity budget with sample size $k$, the resulting optimal temperature increases with $k$, turning empirical choice into a theorem. The optimum still needs the unknown verifier potential, which we estimate using two design approaches. In the online setting, our J-EOS algorithm jointly tracks the two constraint multipliers and learns the verifier potential from binary pass/ fail feedback alone. In the offline setting, we show that when the model is calibrated, its log-probability tracks correctness up to an affine map. Under this condition, the tilt collapses into the exponent and the optimum reduces to ordinary temperature sampling. Calibrated Temperature Decoding (CTD) then fits one scalar offline and samples with zero additional inference-time overhead relative to temperature sampling. Building on these decision rules, we design a practical decoding algorithm for code generation and evaluate it alongside CTD on benchmarks including HumanEval and MBPP, showing improvements over standard and adaptive decoding baselines.