Prism improves language model output diversity and quality together
PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation
Machine Learning
Summary
Large language models often get stuck repeating familiar ideas instead of exploring new ones, which limits creativity and learning. The authors propose PRISM, a way of training models to produce more varied yet correct responses by showing them many diverse examples and tweaking certain parts of their processing to encourage variety. This approach helps the model balance quality with diversity, making it more likely to find rare but better answers and improving tasks like writing stories, math, and coding. PRISM also supports future learning steps by keeping output variety during reinforcement learning.
What this means in practice
- •For software developers: Generate more varied and high-quality code snippets by training language models with PRISM to reduce repetitive outputs and discover better solutions.
- •For content creators: Improve story and text generation tools by integrating PRISM-trained models that maintain diversity, enabling more creative and engaging outputs.
Authors
Qixin Xu, Zihan Zhu, Xiaojiang Liu, Bohan Lyu
Abstract
Large language models (LLMs) exhibit mode collapse, where their output distribution concentrates on familiar semantic modes. When diversity disappears, repeated sampling becomes semantic repetition rather than discovery, making rare but stronger solutions unlikely to surface and leaving self-evolution and continual learning difficult to sustain. Most remedies intervene only at inference time, shaping sam- ples without making diversity a model capability. We propose PRISM, a training-time framework that internalizes diversity within the model. PRISM combines (i) submodular curation that pairs each prompt with diverse-yet-correct multi-teacher solutions, and (ii) stochastic embedding modulation, which turns special-token perturbations into implicit signals for sampling distinct outputs. A theoretical analysis establishes the stability of this mechanism. On WritingPrompts and BookMIA, PRISM moves the quality-diversity frontier outward, achieving the best generation quality in all model-benchmark settings while maintaining the strongest or near-strongest semantic diversity across two open-weight model families. It also improves sampling efficiency on math and code benchmarks. Beyond inference, PRISM yields a stronger foundation for reinforcement learning, sustaining diverse rollouts and mitigating the diversity collapse commonly observed during RL training.