Fine-tuning large language models improves output diversity calibration
Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs
Artificial Intelligence
Summary
Large language models sometimes give answers that are too similar or too different compared to what real people might say. This happens because of how the model is trained and the data used to adjust it. The researchers show that when you fine-tune these models with enough examples, their responses become more like the diversity of the real-world answers the data represents. They also provide mathematical tools to understand why this happens and test their ideas with different models and data types, including human surveys and code examples.
What this means in practice
- •For machine learning engineers: Improve tuning procedures to better match the diversity of user or data-generated responses with large language models.
- •For software development teams: Use fine-tuned language models on code datasets like CodeNet to generate program variations that closely reflect human solutions.
Authors
Kirill Skobelev, Eric Fithian, X. Y. Han
Abstract
Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse, or its opposite, occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently from the same fixed prompt coincide (collide), or their expected similarity under a kernel. We derive a bias-variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback-Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: small transformers on synthetic languages, four LLMs fine-tuned on human surveys, and these LLMs fine-tuned on CodeNet, a dataset of human code solutions. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution.