Transformer models mirror traditional features in multilingual readability tasks
Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment
Computation and Language
Summary
Assessing how hard a text is to read often relies on either classic methods using linguistic features or newer AI models called transformers. This paper looks at whether transformer models understand the same language features as traditional methods in multiple languages. The authors find transformers do pick up on features like sentence length and vocabulary variety that affect readability. However, how well they track these features depends on the model type, language, and the model’s internal layers.
What this means in practice
- •For natural language processing teams: Improve multilingual text difficulty evaluations by integrating transformer insights with linguistic feature analysis.
- •For language assessment developers: Develop readability tools that combine traditional linguistics and transformer model outputs for better language-specific accuracy.
Authors
Joshua Wong, Chris Tanner
Abstract
Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties. This matters because readability labels are subjective and rater-dependent, so high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty. We test whether transformers internalize the same features as traditional models across Arabic, English, French, Hindi, and Russian using the ReadMe++ dataset. Shapley Additive Explanations (SHAP) identify the features driving traditional classifiers, which we then use as TCAV concept sets to probe multilingual XLM-R and language-specific encoders. Transformers recover surface-length, syntactic, and lexical-diversity signals, and reflect the ordinal CEFR structure of the traditional models. Alignment varies by model family, language, and layer, with language-specific encoders tracking traditional models more clearly than XLM-R. High linear separability does not always imply directional influence, limiting linear probing for count-based readability features.