Language models use small word sets in poetry despite large vocabularies

Language Models are Lazy Poets: Creative Mode Collapse at Scale

Computation and Language

Summary

Large language models can create poems but tend to repeat the same words, phrases, and rhymes over and over, even though they know many more. The authors studied over a million poems generated in many styles and found that this 'laziness' varies by model and type of poem. Bigger models tend to use a larger rhyme vocabulary but still rely on a limited subset of language. This means that current models are less creative in poetry than one might expect given their capabilities.

What this means in practice

  • For creative software developers: Improve poetry generation tools by identifying and addressing word and rhyme repetition patterns in large language models.
  • For language ai engineers: Design better language models by analyzing how model size affects poetic diversity and reduce over-reliance on common phrases during training.

Authors

Jacob A. Matthews, Sil Hamilton

Abstract

While aligned large language models (LLMs) can successfully complete many creative tasks, their outputs tend to be highly self-similar. Though solutions to this problem of "mode collapse" have been proposed, comprehensive large-scale surveys of the phenomenon in specific textual genres are still lacking. We address this gap through an extensive treatment of LLM mode collapse in poetry. Across 15 poetic forms, 248 subject matter tags, and 16 open-weight models, we generate over 1.1 million poems. To our knowledge, this constitutes the largest corpus of LLM-generated poetry to date. At million-poem scale, we find that LLM poetic vocabularies and rhyming abilities are substantial, but severely underutilized. In other words, LLMs are "lazy" poets: despite possessing large poetic inventories, LLMs over-rely on a small number of words, opening phrases, and rhymes. Further, we find that lexical homogenization differs across models and poetic forms, noting observed rhyming vocabulary is correlated with model size.