Semantic complexity of text measured using transformer attention
Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention
Computation and LanguageArtificial Intelligence
Summary
Measuring how complex a text's meaning is can be tricky. The paper defines semantic complexity as how much information is needed to uniquely describe the meaning of a text. The authors introduce SCAPE-T, a new way to measure this complexity by looking inside attention patterns of transformer models, a type of AI used for language. Their method works better than existing ones that rely on word or sentence features, and it can tell apart texts by how conceptually complex they are.
What this means in practice
- •For natural language processing teams: Improve systems that need to estimate text meaning complexity by integrating SCAPE-T for more nuanced semantic analysis beyond surface features.
- •For digital content platforms: Refine content recommendation algorithms by using semantic complexity scores to better match user comprehension levels and interests.
- •For ai writing assistance providers: Create advanced feedback tools that assess semantic complexity to help users write clearer or more sophisticated texts.$Commercial implications: Enables selling AI writing tools that provide detailed advice on semantic complexity tailored to user goals.
Authors
Maxim Khovansky, Lev Mukhanov
Abstract
Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.