Enriching text structure does not improve coherence detection accuracy
Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures
Computation and LanguageArtificial Intelligence
Summary
Sometimes, computer programs that write sentences can make grammatically correct but confusing or contradictory text. This paper finds that adding extra information about sentence structure and storytelling connections does not help current models detect when text is incoherent. In fact, using only plain text worked better because the extra structural information didn't fit well with how these models work. They also showed that checking how coherent a text is might help spot misleading or false information in online content.
What this means in practice
- •For content moderation teams: Use coherence assessment in text to help detect misleading content in social media posts without retraining models.
- •For fact-checking organizations: Incorporate coherence metrics as a fast proxy to flag potentially deceptive or contradictory claims in online texts.
Authors
Victor Mazzotti, Luiz Pereira, Marina Bitencourt dos Santos, Helena Maia, Carlos Caetano, Nádia Felix, Sandra Avila
Abstract
Recent advances in large language models have transformed human-computer interaction. Despite their fluency, these models often produce texts that are grammatically correct but semantically incoherent, containing contradictions or disruptions in logical flow. This work investigates whether enriching text with syntactic and rhetorical information can improve incoherence prediction. Our experiments and analysis show that plain texts achieved higher accuracy because the added information was structurally and syntactically incompatible with the language model's architecture. Additionally, to demonstrate the practical importance of coherence assessment, we performed zero-shot experiments on a Brazilian disinformation dataset, suggesting that textual coherence can serve as a proxy for detecting misleading content. Code and models are available at https://github.com/ittozzamV/cohereclassifier.