User edits reveal how people explore stories using large language models
The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation
Computation and Language
Summary
People writing stories with AI often change their instructions many times to try out new characters, plots, or settings. The authors looked at a huge collection of these changing instructions from chatbots to understand how people explore different story ideas. They created tools to categorize the types of changes users make and to analyze how people navigate through different story options. Their work also shows how to make better tests for story-generating AI by including this back-and-forth editing process. They caution that some of the story content in their data includes sensitive material.
What this means in practice
- •For ai developers: Build story generation models that better reflect how users iteratively explore and refine narratives with multiple prompt edits.
- •For interactive fiction designers: Create branching story experiences that mimic real user storytelling behavior tracked via prompt editing patterns.
Authors
Advait Deshmukh, Nora Benedict, Melanie Walsh, Maria Antoniak
Abstract
Large language models (LLMs) have changed the way people engage with stories. Drawing on public chatbot logs, we can see that when users generate stories, they iteratively edit their prompts to explore narrative possibilities, adjusting characters, redirecting plots, and swapping fictional universes. As aggregated data, these prompts represent rich traces of creative preference at scale. Yet story generation evaluation benchmarks rely on static, one-shot prompts that cannot capture this exploratory behavior. In this work, we study how users revise consecutive story prompts in the wild. Using a dataset of naturally occurring user-chatbot conversations, we construct WildStories, a sample of 275,635 story generation prompts (labeled with story format, prompt components, and explicitness), and WildEdits, a collection of 24,291 edit trees that model how users iteratively edit base story prompts and explore branching story possibilities. From these trees we develop a framework of edit types crossing four directions (adding, removing, changing, and extending) with fourteen targets (e.g., plot, character, genre). We then use our datasets and this framework to analyze user behavior in navigating narrative space via LLMs. Finally, we show how automated permutations based on the framework can be used for story generation benchmarking. Content Warning: This paper works with "wild" chatbot logs, which often include toxic and sexually explicit themes.