Temperature changes affect large language model answer accuracy significantly
Temperature Fragility and the Conditional Benefits of Truncation Sampling
Computation and LanguageMachine Learning
Summary
Large language models use a 'temperature' setting to decide how creative or predictable their word choices are when generating text. The authors studied how this setting and certain filters called truncation samplers impact the quality of answers across several models. They found that raising temperature sometimes causes big drops in accuracy, while truncation samplers only help when high temperatures make performance worse. When accuracy is stable across temperatures, these filters do not improve results.
What this means in practice
- •For machine learning engineers: Tune temperature and truncation settings carefully when deploying large language models to avoid accuracy losses in sensitive applications.
- •For ai platform operators: Use the findings to set decoding defaults that maintain performance stability across different language models and tasks.
Authors
Francesco La Rosa
Abstract
Large language models generate text by sampling each token from a predicted distribution, and a temperature parameter sets how far the draw strays from the most probable tokens. Truncation samplers such as top-p and min-p discard the least probable tokens before the draw, so that sampling at high temperature stays coherent. Their reported accuracy gains come from temperatures of 1.5 to 3, while the defaults of deployed systems cluster between 0.6 and 1.0. Whether they change accuracy at those defaults, and for which models, has not been measured. We test thirteen open-weight models on GSM8K and MMLU-Pro at temperatures 0.7, 1.0, and 1.3 in one controlled pipeline, ten of them under eight decoding configurations. Six of the thirteen models lose 17 to 38 accuracy points on MMLU-Pro between 0.7 and 1.3, and the other seven lose at most 10. The lost accuracy comes from generations that run to the token limit or never state an answer. These results suggest that truncation samplers improve accuracy primarily when higher temperatures substantially degrade model performance. Where accuracy remains stable across temperatures, none of the tested truncation samplers improves on plain temperature sampling.