Mode-conditioned learning improves diversity and quality in large language models
Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning
Computation and LanguageArtificial IntelligenceMachine LearningMultiagent Systems
Summary
Large language models often lose variety in their answers after alignment training, which limits creativity and exploration. The authors propose a new method called MoDA, which trains a model to produce high-quality but diverse responses by assigning different 'roles' that compete to generate unique outputs. This prevents the model from repeating the same ideas and ensures the quality stays high. The method was tested on various tasks and showed significant improvements in both creativity and correctness compared to other approaches.
What this means in practice
- •For content generation teams: Enable AI writing assistants to produce more diverse and creative outputs without quality loss for applications like story writing and brainstorming.$Commercial implications: This method allows companies building AI creative tools to offer more varied and engaging content, improving user experience and market value.
- •For ai system developers: Improve large language model deployment by integrating MoDA to maintain output diversity and accuracy in customer service or research support systems.
Authors
Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques
Abstract
A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.