Large language models struggle to fill narrative gaps reliably
Evaluating Whether LLMs Can Reliably Connect the DOTs?
Computation and Language
Summary
Putting together pieces of a story when parts are missing can be really tricky, especially if the information is jumbled or incomplete. The authors studied how well popular language AI models can fill in missing sentences in different types of stories. Surprisingly, they found that bigger models aren't always better, and some smaller models did the job more smoothly. They also discovered that making models explain their reasoning doesn’t help much, but the type and length of the story matters a lot for how well the models do.
What this means in practice
- •For content moderation teams: Evaluate how reliably large language models can reconstruct missing parts of user-generated narratives for better content understanding.
- •For digital archivists: Improve tools for reconstructing fragmented historical or news articles by understanding narrative infilling challenges found in different domains.
Authors
Eftekhar Hossain, John Salvador, Santu Karmaker
Abstract
Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct missing spans within a broader storyline, commonly referred to as text infilling, while preserving consistency with both the local context and the global storyline. Despite using text infilling as a pre-training objective in many Large Language Models (LLMs), their actual performance on real-world narrative infilling remains underexplored. In this paper, we address this gap by introducing a multi-domain benchmark of ~9.2K instances for narrative infilling, constructed by masking one to three sentences across four narrative types: encyclopedic text, commonsense stories, news articles, and visual narratives. Using this benchmark, we evaluate 20 instruction-tuned open-source LLMs ranging from 1.5B to 70B parameters across varying levels of instruction specificity and reasoning guidance. Outputs are assessed using standard automatic metrics and a qualitative framework covering five narrative dimensions. Results show that model scale does not reliably predict infilling quality: Gemma-2-2B achieves the highest qualitative score (4.02/5), outperforming models over ten times larger, including DeepSeek-Qwen-32B (3.77/5, 6.6%) and LLaMA-3.3-70B (3.71/5, 8.3%). We further find that explicit reasoning offers limited benefits as chain-of-thought reasoning yields only a marginal improvement (+0.6%). Additionally, short narratives and domain characteristics emerge as stronger predictors of task difficulty than infill position alone for narrative infilling in current LLMs.