Large language models struggle to identify overlapping and conflicting narrative parts
Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?
Computation and Language
Summary
Understanding stories or texts from multiple points of view is hard because parts of the story might agree, disagree, or be unique. The authors created a test to see if language AI models can pick out these overlapping, conflicting, and unique parts between two stories. They found that AI models do well with unique parts but have trouble identifying overlaps and conflicts, especially with smaller models. Training models specifically for this task helps, but even the best models still find it challenging.
What this means in practice
- •For fact-checkers: Improve tools to automatically identify agreements, contradictions, and unique claims across multiple news stories or reports.
- •For legal document analysts: Assist in extracting conflicting or unique clauses from different versions of legal texts for comparison and review.
Authors
Eftekhar Hossain, Santu Karmaker
Abstract
Understanding multi-perspective alternative narratives requires identifying how their information agrees, conflicts, or differs across sources. Existing work on cross-text relations largely focuses on categorizing relations between predefined text pairs, such as entailment or contradiction, rather than directly extracting such information from full narratives. To address this gap, we introduce Overlap-Unique-Conflict (OUC) extraction, a cross-narrative task that extracts all overlapping, conflicting, and unique clauses from two narratives. To support this study, we construct a benchmark of approximately 22K narrative pairs and 140K OUC instances spanning factual, argumentative, and political discourse. Evaluating 14 open-source LLMs (0.6B-35B), we find that unique information is far easier to extract than overlap and conflict: the strongest model, Gemma-4-31B, reaches only 61.13% F1-score on overlap and 48.58% on conflict, against more than 75% on unique. Further diagnostic analysis reveals that this difficulty does not stem from relation recognition alone, but rather from a failure to pair and extract the corresponding clauses from full narratives, especially in smaller models. Nevertheless, learning these extractions with task-specific supervision narrows the gap considerably: a fine-tuned Qwen-3-8B gains 15-28% absolute over its baseline and surpasses models roughly four times its size (e.g., Qwen-3.6-35B) on several tasks. Even so, overlap and conflict remain well below satisfactory, leaving cross-narrative clause extraction an open challenge.