Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
2026-05-07 • Machine Learning
Machine LearningDiscrete MathematicsEmerging Technologies
AI summaryⓘ
The authors studied how large language models (LLMs) are ranked using human comparisons across many languages and found that a single global ranking can be misleading because people's opinions vary a lot depending on language and task. They showed that grouping comparisons by language greatly improves agreement and creates clearer rankings. To handle this diversity, they introduced a new method called (λ, ν)-portfolios, which selects small sets of models that reliably satisfy most users. Their approach better represents diverse preferences and also helps identify gaps in data fairness when applied to other tasks.
Large Language ModelsBradley-Terry ModelPairwise ComparisonELO RankingSet Cover ProblemVC DimensionFairness in Machine LearningEnsemble MethodsHuman FeedbackModel Portfolios
Authors
Jai Moondra, Ayela Chughtai, Bhargavi Lanka, Swati Gupta
Abstract
Ranking LLMs via pairwise human feedback underpins current leaderboards for open-ended tasks, such as creative writing and problem-solving. We analyze ~89K comparisons in 116 languages from 52 LLMs from Arena, and show that the best-fit global Bradley-Terry (BT) ranking is misleading. Nearly 2/3 of the decisive votes cancel out, and even the top 50 models according to the global BT ranking are statistically indistinguishable (pairwise win probabilities are at most 0.53 within the top 50 models). We trace this failure to strong, structured heterogeneity of opinions across language, task, and time. Moreover, we find an important characteristic - *language* plays a key role. Grouping by language (and families) increases the agreement of votes massively, resulting in two orders of magnitude higher spread in the ELO scores (i.e., very consistent rankings). What appears as global noise is in fact a mixture of coherent but conflicting subpopulations. To address such heterogeneity in supervised machine learning, we introduce the framework of $(λ, ν)$-portfolios, which are small sets of models that achieve a prediction error at most $λ$, "covering" at least a $ν$ fraction of users. We formulate this as a variant of the set cover problem and provide guarantees using the VC dimension of the underlying set system. On the Arena data, our algorithms recover just 5 distinct BT rankings that cover over 96% of votes at a modest $λ$, compared to the 21% coverage by the global ranking. We also provide a portfolio of 6 LLMs that cover twice as many votes as the top-6 LLMs from a global ranking. We further construct portfolios for a classification problem on the COMPAS dataset using an ensemble of fairness-regularized classification models and show that these portfolios can be used to detect blind spots in the data, which might be of independent interest to policymakers.