Calibrated metrics match judges in multilingual style text evaluation
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Computation and Language
Summary
Measuring how well computer programs judge text quality can be tricky, especially across different languages and styles. The authors created a way to better evaluate these judging tools by teaching them to predict human scores more accurately. They tested this method on a large, multilingual set of texts with many human ratings, showing that even simple measurements can perform as well as more complex AI judges when properly calibrated. They also learned what factors affect how well these judges work and when combining different judges is beneficial.
What this means in practice
- •For multilingual nlp engineers: Improve evaluation of style-personalized text generators across many languages using calibrated metrics that better align with human judgment.
- •For text analytics platform developers: Build more reliable quality assessment tools by combining multiple metrics and calibrating them to human scores for diverse text styles and languages.$Commercial implications: Enables creation of superior multilingual text quality scoring services that can be marketed to content moderation and generation companies.
Authors
Anubhav Jangra, Adrian de Wynter
Abstract
Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.