AI summaryⓘ
The authors created ALEE, a new way to test how well computer models understand the meaning of sentences across many languages and longer texts. They build on an earlier method called Sentence Smith by using a tool called Abstract Meaning Representations to make small, precise changes in English sentences and then pair them with translations. This helps them check models in over 275 languages and find where the models struggle, especially with languages that have less training data. Their tests revealed big differences in model performance depending on language and sentence length, showing that current models still have trouble with many languages. The authors made their ALEE tool publicly available for others to use.
Text embeddingsSemantic similarityCross-lingual evaluationAbstract Meaning Representation (AMR)Minimal pairsParallel datasetsSubword tokenizationLow-resource languagesSentence SmithSemantic representation
Authors
Andrianos Michail, Stylianos Psychias, Michelle Wastl, Simon Clematide, Rico Sennrich, Juri Opitz
Abstract
Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval