Language models struggle to tell stories out of order clearly
Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation
Computation and LanguageArtificial Intelligence
Summary
Stories can be told in the order events happen or in a mixed-up order, which is called nonlinear storytelling. The authors collected 83 stories and tested if language models (AI that writes text) can tell these nonlinear stories well. They found that models usually tell stories more in order than humans do and don’t tell nonlinear stories as clearly as when told in order. People also found the models’ nonlinear stories harder to follow. The research shows that just asking models to reorder stories doesn’t make them tell those stories effectively.
What this means in practice
- •For creative writing software developers: Improve story generation tools by understanding language models’ limits in nonlinear storytelling and testing better narrative structures.
- •For game narrative designers: Use insights on language models’ performance to guide AI-driven nonlinear story creation in interactive games.
Authors
Syed Rifat Raiyan, Md. Tariquzzaman, Md Farhan Ishmam, Tenghao Huang
Abstract
Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.