Framework measures meaning and coverage differences in text comparisons
A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation
Computation and LanguageArtificial Intelligence
Summary
When two texts describe the same thing, usual methods that compare words may miss important differences in meaning or focus. The authors created a new way to check how well two texts match in meaning and what parts they cover, calling these measures Semiotic Fidelity and Semiotic Coverage. They found that texts often don't cover each other's full meaning, especially when generated by AI with more randomness. This method helps understand how closely generated text matches human-like meaning.
What this means in practice
- •For natural language developers: Assess the meaningful alignment between generated texts and reference texts to improve evaluation beyond word-based metrics.
- •For content quality teams: Identify how well automatically generated content captures intended meanings and references compared to human-written content.
Authors
Lorenzo Zangari, Davide Picca
Abstract
When two texts describe the same expression, standard metrics based on lexical overlap or whole-text similarity may fail to detect meaningful differences in how that expression is framed. We propose a framework to evaluate semiotic alignment between texts, where a semiotic profile encompasses both the contextual meaning and the discourse references made salient by a text. Our approach yields two scores, Semiotic Fidelity and Semiotic Coverage, estimating how much of one text's profile is supported by the other and how much of the other's profile it recovers. Experiments show that coverage is typically lower than fidelity, and that alignment between LLMs and human-curated data is highest at low sampling temperatures, while higher temperatures reduce this alignment.