High-level text preprocessing improves similarity accuracy in discursive documents
High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration
Computation and LanguageInformation RetrievalMachine Learning
Summary
Documents like philosophical essays often include lots of discussion about other ideas, which can confuse methods that try to find how similar documents are based on their words. The authors introduce a new way to clean up the text before comparison by separating the main claims from the surrounding discussion. This helps the similarity scores reflect true agreement instead of just shared vocabulary. They tested this on philosophy texts using advanced language models and found the approach consistently lowered inflated similarity scores, showing clearer differences between ideas.
What this means in practice
- •For legal analysts: Improve similarity comparisons of legal documents where texts critique or discuss other cases by isolating actual claims from discursive parts.
- •For policy researchers: Enhance analysis of policy documents that reference and debate other policies by reducing inflated similarity scores due to discursive language.
Tested on one dataset.
Authors
Mehmet Murat Albayrakoglu, Mehmet Nafiz Aydin
Abstract
Semantic Textual Similarity (STS) methods assume that a document's lexical content faithfully represents what it asserts. This assumption fails for discursive documents that discuss, compare, critique, and contextualize other positions in the process of articulating their own. The result is semantic diffusion: similarity scores between documents are inflated by vocabulary acquired through discursive engagement rather than substantive alignment. Standard Natural Language Processing (NLP) preprocessing (tokenization, stopword removal, stemming, lemmatization) cannot address this problem because it operates at the lexical level, treating all content identically regardless of its discursive function. This paper introduces high-level text preprocessing: a systematic, rule-based intervention applied before the standard preprocessing pipeline to isolate each document's actual claim from its discursive structure. We propose 12 rules, each with an explicit rationale, and demonstrate their effect on an encyclopedic philosophical corpus: three entries from the Stanford Encyclopedia of Philosophy (virtue ethics, deontological ethics, and consequentialism). A three-phase experiment using eight Transformer-based STS models shows that preprocessing reduces centroid cosine similarity scores across all three theory pairs, with 23 of 24 model-pair comparisons showing the expected decrease and cross-model agreement ranging from 7-1 to 8-0. We introduce the semantic diffusion index (SDI), a per-document metric for assessing the semantic reorientation between a document's raw and high-level preprocessed representations. Although the framework is demonstrated using philosophical texts, it potentially addresses a domain-agnostic problem applicable to legal texts, policy documents, academic articles, and any genre in which a discursive approach introduces vocabulary from positions the document does not endorse.