MATCHA: Matching Text via Contrastive Semantic Alignment
2026-05-26 • Computation and Language
Computation and Language
AI summaryⓘ
The authors point out that common methods for checking how similar two texts are, like ROUGE and BERTScore, can give high scores even when the texts actually say opposite things. To fix this, they created MATCHA, a tool that not only looks for agreement with a correct text but also checks for contradictions by comparing against purposely tricky wrong versions. They tested MATCHA on several tasks and found it better matches human judgment than popular scores, especially in spotting mistakes. Their work shows some existing metrics miss important errors, and MATCHA is more accurate at telling correct from incorrect statements.
large language modelstext similarity metricsROUGEBERTScoresemantic agreementcontradiction detectionnatural language inferenceautomatic evaluationcounterfactualsTruthfulQA
Authors
Siran Li, Ece Sena Etoglu, Carsten Eickhoff, Seyed Ali Bahrainian
Abstract
Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g., ROUGE) and embedding-based measures (e.g., BERTScore), often misjudge semantic similarity of documents. Our study shows that both token-overlap metrics and embedding-based metrics routinely assign nearly identical scores to texts that directly contradict each other, thereby potentially masking fundamental errors. We introduce MATCHA, an automatic metric that jointly rewards semantic agreement with a reference and penalizes contradictions. MATCHA employs a dual-view perspective that measures (i) proximity to the gold text and (ii) distance from an adversarially generated counterfactual contradiction. In eight public benchmarks, MATCHA outperforms popular metrics, compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. On the TruthfulQA dataset (i.e., a dataset without a training set, where no embedding-based metrics could locally train on), this improvement in terms of matching texts with a reference reaches 18.38% over ROUGE-L and 20.82% over BERTScore. Both quantitative comparison and qualitative human assessments confirm the efficacy and validity of MATCHA and uncover fundamental weaknesses in pre-existing metrics. Compared with 23 embedding models, including top state-of-the-art ones, used as a metric similar to BERTScore, MATCHA remains the most accurate in distinguishing correct from incorrect statements solely based on a reference. Our code and metric are publicly available (https://github.com/Siran-Li/MATCHA).