Comparing what you analyze matters more than how you analyze it
A Better Metric Will Not Save You: What You Compare Beats How You Compare It
Computation and Language
Summary
When comparing different systems, it's common to use metrics to see how similar they are. This paper finds that the choice of which items or groups you compare has a bigger effect on the comparison result than the specific metric you use. The authors tested this on vowel sounds across many languages and found that changing comparison groups changes results more than changing the number of groups or the metric itself. This means how you decide what to compare is more important than how you measure it.
What this means in practice
- •For speech technology engineers: Select input groupings carefully to improve evaluation consistency when comparing speech recognition or synthesis models on vowel sounds.
- •For natural language processing teams: Adjust how condition sets are chosen when comparing language models to better understand cross-lingual differences in sound representation.
Authors
Noureddine Khaous, Guillaume Wisniewski
Abstract
Representational similarity measures (RSA, CKA, Procrustes) are chosen for the transformations they treat as irrelevant, and all of them take the set of conditions as given. That set is fixed by two decisions, which inputs are admitted and how they are grouped, both taken before the matrix a measure operates on exists: no invariance property reaches them. We measure what they are worth where the arrangement the conditions ought to have is known independently of the systems compared, the acoustic organisation of vowels, across fifteen languages and twenty-seven representations. Changing which categories are compared, at a fixed number of them, moves the gap between two representations by more than twice the effect being measured, while changing how many moves it not at all. In this setting rankings of models are identified under these choices, and rankings across languages are not.